[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:42 dist-test-slave-252r krb5kdc[13186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:42 dist-test-slave-252r krb5kdc[13186](info): set up 2 sockets
Sep 25 02:52:43 dist-test-slave-252r krb5kdc[13186](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 02:52:44.109294 13179 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:52:44 dist-test-slave-252r krb5kdc[13186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:52:44.258967 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:39931
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:39931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:44.390084 13199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:44.390338 13199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:44.390409 13199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:44.394233 13199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:44.394312 13199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:44.394345 13199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:44.394382 13199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:44.394414 13199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:44.399554 13199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:39931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:39931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:44.400604 13199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:44.401697 13199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:52:44.410897 13199 server_base.cc:1034] running on GCE node
W20260925 02:52:44.411127 13207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:44.412168 13204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:44.412432 13205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:44.413043 13199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:44.414153 13199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:44.415333 13199 hybrid_clock.cc:648] HybridClock initialized: now 1790304764415317 us; error 74 us; skew 500 ppm
Sep 25 02:52:44 dist-test-slave-252r krb5kdc[13186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304764, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:44.419551 13199 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:52:44.420847 13199 webserver.cc:460] Webserver started at http://127.12.222.254:46477/ using document root <none> and password file <none>
I20260925 02:52:44.421696 13199 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:44.421828 13199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:44.422230 13199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:44.425704 13199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "323f2ceda447482095061592cc027f34"
format_stamp: "Formatted at 2026-09-25 02:52:44 on dist-test-slave-252r"
server_key: "b25683cbdfde74bf5169260ee786bcb0"
server_key_iv: "a10630acebeea195d11adb821ae4ef41"
server_key_version: "encryptionkey@0"
I20260925 02:52:44.426568 13199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "323f2ceda447482095061592cc027f34"
format_stamp: "Formatted at 2026-09-25 02:52:44 on dist-test-slave-252r"
server_key: "b25683cbdfde74bf5169260ee786bcb0"
server_key_iv: "a10630acebeea195d11adb821ae4ef41"
server_key_version: "encryptionkey@0"
I20260925 02:52:44.432667 13199 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 02:52:44.436353 13214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:44.438359 13199 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:52:44.438586 13199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "323f2ceda447482095061592cc027f34"
format_stamp: "Formatted at 2026-09-25 02:52:44 on dist-test-slave-252r"
server_key: "b25683cbdfde74bf5169260ee786bcb0"
server_key_iv: "a10630acebeea195d11adb821ae4ef41"
server_key_version: "encryptionkey@0"
I20260925 02:52:44.438750 13199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:44.463258 13199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:44.466022 13199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:44.466217 13199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:44.474263 13199 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:39931
I20260925 02:52:44.474308 13266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:39931 every 8 connection(s)
I20260925 02:52:44.475133 13199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:44.478153 13267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:44.484131 13267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34: Bootstrap starting.
I20260925 02:52:44.485142 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 13199
I20260925 02:52:44.485251 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:52:44.486469 13179 external_mini_cluster.cc:1428] Setting key 987ca9e1f5f45e957b430c24cdac969a
I20260925 02:52:44.486948 13267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:44.487692 13267 log.cc:824] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:44.490227 13267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34: No bootstrap required, opened a new log
I20260925 02:52:44.499902 13267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323f2ceda447482095061592cc027f34" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39931 } }
I20260925 02:52:44.500177 13267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:44.500295 13267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 323f2ceda447482095061592cc027f34, State: Initialized, Role: FOLLOWER
I20260925 02:52:44.501138 13267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323f2ceda447482095061592cc027f34" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39931 } }
I20260925 02:52:44.501362 13267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:44.501488 13267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:44.501626 13267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:44.502704 13267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323f2ceda447482095061592cc027f34" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39931 } }
I20260925 02:52:44.503159 13267 leader_election.cc:302] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 323f2ceda447482095061592cc027f34; no voters:
I20260925 02:52:44.503540 13267 leader_election.cc:288] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:44.503655 13272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:44.503832 13272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34 [term 1 LEADER]: Becoming Leader. State: Replica: 323f2ceda447482095061592cc027f34, State: Running, Role: LEADER
I20260925 02:52:44.504213 13272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323f2ceda447482095061592cc027f34" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39931 } }
I20260925 02:52:44.505118 13267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:44.505765 13274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 323f2ceda447482095061592cc027f34. Latest consensus state: current_term: 1 leader_uuid: "323f2ceda447482095061592cc027f34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323f2ceda447482095061592cc027f34" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39931 } } }
I20260925 02:52:44.505903 13274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:44.506889 13279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:44.507623 13273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "323f2ceda447482095061592cc027f34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323f2ceda447482095061592cc027f34" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39931 } } }
I20260925 02:52:44.507717 13273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:44.510619 13279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 02:52:44 dist-test-slave-252r krb5kdc[13186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:44.517130 13279 catalog_manager.cc:1347] Generated new cluster ID: 6444b33e25084d8c990dc7454803018c
I20260925 02:52:44.517248 13279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:44.517450 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:44.489478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33928 (local address 127.12.222.254:39931)
0925 02:52:44.489918 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:44.489937 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:44.491248 (+ 1311us) server_negotiation.cc:407] Connection header received
0925 02:52:44.499551 (+ 8303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:44.499567 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:44.499862 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:44.500109 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:44.503315 (+ 3206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:44.504656 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:44.506084 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:44.506757 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:44.506928 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:44.511603 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:44.511609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:44.511622 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:44.511648 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:44.513792 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:44.514450 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:44.514455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:44.514460 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:44.514622 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:44.514995 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:44.514998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:44.515000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:44.515551 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:44.515731 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:44.516282 (+ 551us) server_negotiation.cc:299] Negotiation successful
0925 02:52:44.516524 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":138,"threads_started":1}
I20260925 02:52:44.524572 13279 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:44.525467 13279 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:44.537567 13279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 323f2ceda447482095061592cc027f34: Generated new TSK 0
I20260925 02:52:44.538255 13279 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
I20260925 02:52:44.571388 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:39931
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:44.716974 13293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:44.717190 13293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:44.717227 13293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:44.721817 13293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:44.721868 13293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:44.721922 13293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:52:44.726469 13293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:39931
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:44.727342 13293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:44.728209 13293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:44.737041 13299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:44.737052 13298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:44.737226 13293 server_base.cc:1034] running on GCE node
W20260925 02:52:44.737538 13301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:44.737907 13293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:44.738492 13293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:44.739663 13293 hybrid_clock.cc:648] HybridClock initialized: now 1790304764739654 us; error 28 us; skew 500 ppm
Sep 25 02:52:44 dist-test-slave-252r krb5kdc[13186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304764, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:44.742295 13293 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:52:44.743140 13293 webserver.cc:460] Webserver started at http://127.12.222.193:34505/ using document root <none> and password file <none>
I20260925 02:52:44.743619 13293 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:44.743697 13293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:44.743907 13293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:44.745827 13293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "e0d3238a7a2f47d08b0d833a992d2aa4"
format_stamp: "Formatted at 2026-09-25 02:52:44 on dist-test-slave-252r"
server_key: "f04264d6432307baa9063e5da074d3ea"
server_key_iv: "65e49c8df3d31fc438de7940079cc425"
server_key_version: "encryptionkey@0"
I20260925 02:52:44.746392 13293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "e0d3238a7a2f47d08b0d833a992d2aa4"
format_stamp: "Formatted at 2026-09-25 02:52:44 on dist-test-slave-252r"
server_key: "f04264d6432307baa9063e5da074d3ea"
server_key_iv: "65e49c8df3d31fc438de7940079cc425"
server_key_version: "encryptionkey@0"
I20260925 02:52:44.749821 13293 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:52:44.752022 13309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:44.753108 13293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:44.753229 13293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "e0d3238a7a2f47d08b0d833a992d2aa4"
format_stamp: "Formatted at 2026-09-25 02:52:44 on dist-test-slave-252r"
server_key: "f04264d6432307baa9063e5da074d3ea"
server_key_iv: "65e49c8df3d31fc438de7940079cc425"
server_key_version: "encryptionkey@0"
I20260925 02:52:44.753336 13293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:44.775695 13293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:44.778192 13293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:44.778357 13293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:44.778887 13293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:52:44.779788 13293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:44.779856 13293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:44.779924 13293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:44.779965 13293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:44.789400 13293 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:35019
I20260925 02:52:44.789422 13423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:35019 every 8 connection(s)
I20260925 02:52:44.790198 13293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:52:44.797915 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 13293
I20260925 02:52:44.798034 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:52:44.798283 13179 external_mini_cluster.cc:1428] Setting key da684efc69092d90832c14778a5ef9c0
Sep 25 02:52:44 dist-test-slave-252r krb5kdc[13186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304764, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:44.803300 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:44.791920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:48531 (local address 127.12.222.254:39931)
0925 02:52:44.792079 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:44.792084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:44.793123 (+ 1039us) server_negotiation.cc:407] Connection header received
0925 02:52:44.793874 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:44.793877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:44.793924 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:44.794000 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:44.795138 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:44.795740 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:44.796976 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:44.797224 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:44.797303 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:44.799895 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:44.799898 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:44.799900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:44.799920 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:44.801435 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:44.801902 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:44.801906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:44.801908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:44.801988 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:44.802298 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:44.802302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:44.802304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:44.802467 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:44.802565 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:44.803027 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 02:52:44.803142 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:52:44.804131 13426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:44.792286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:39931 (local address 127.12.222.193:48531)
0925 02:52:44.792973 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:44.793009 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:44.793709 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:44.794153 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:44.794159 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:44.794469 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:44.794990 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:44.795004 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:44.795850 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:44.795853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:44.796844 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:44.796852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:44.797352 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:44.797354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:44.797507 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:44.798197 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:44.798213 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:44.799746 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:44.801549 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:44.801554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:44.801566 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:44.801804 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:44.802075 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:44.802078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:44.802081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:44.802209 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:44.802565 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:44.802572 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:44.802785 (+ 213us) client_negotiation.cc:769] Sending connection context
0925 02:52:44.802976 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 02:52:44.803256 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":115,"threads_started":1}
I20260925 02:52:44.805330 13424 heartbeater.cc:347] Connected to a master server at 127.12.222.254:39931
I20260925 02:52:44.805578 13424 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:44.806141 13424 heartbeater.cc:511] Master 127.12.222.254:39931 requested a full tablet report, sending...
I20260925 02:52:44.807933 13231 ts_manager.cc:194] Registered new tserver with Master: e0d3238a7a2f47d08b0d833a992d2aa4 (127.12.222.193:35019)
WARNING: no policy specified for kudu/127.12.222.194@KRBTEST.COM; defaulting to no policy
I20260925 02:52:44.809105 13231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:48531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
I20260925 02:52:44.824580 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:39931
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:44.934958 13429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:44.935164 13429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:44.935226 13429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:44.939051 13429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:44.939097 13429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:44.939188 13429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:52:44.944108 13429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:39931
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:44.944965 13429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:44.945788 13429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:44.954591 13435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:44.954689 13437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:44.954771 13434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:44.954799 13429 server_base.cc:1034] running on GCE node
I20260925 02:52:44.955221 13429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:44.955734 13429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:44.956869 13429 hybrid_clock.cc:648] HybridClock initialized: now 1790304764956852 us; error 27 us; skew 500 ppm
Sep 25 02:52:44 dist-test-slave-252r krb5kdc[13186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304764, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:44.959517 13429 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:52:44.960296 13429 webserver.cc:460] Webserver started at http://127.12.222.194:43865/ using document root <none> and password file <none>
I20260925 02:52:44.960927 13429 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:44.961022 13429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:44.961222 13429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:44.963825 13429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "e1d3d1230a1746d4889c8e22fb49bbbc"
format_stamp: "Formatted at 2026-09-25 02:52:44 on dist-test-slave-252r"
server_key: "c587b1f20840aa3e62b2a54d93407459"
server_key_iv: "4887bd6678171734b2704cd11aa9e35a"
server_key_version: "encryptionkey@0"
I20260925 02:52:44.964517 13429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "e1d3d1230a1746d4889c8e22fb49bbbc"
format_stamp: "Formatted at 2026-09-25 02:52:44 on dist-test-slave-252r"
server_key: "c587b1f20840aa3e62b2a54d93407459"
server_key_iv: "4887bd6678171734b2704cd11aa9e35a"
server_key_version: "encryptionkey@0"
I20260925 02:52:44.968890 13429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 02:52:44.970988 13444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:44.972102 13429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:52:44.972213 13429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "e1d3d1230a1746d4889c8e22fb49bbbc"
format_stamp: "Formatted at 2026-09-25 02:52:44 on dist-test-slave-252r"
server_key: "c587b1f20840aa3e62b2a54d93407459"
server_key_iv: "4887bd6678171734b2704cd11aa9e35a"
server_key_version: "encryptionkey@0"
I20260925 02:52:44.972363 13429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:44.982985 13429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:44.985366 13429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:44.985571 13429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:44.986060 13429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:52:44.986773 13429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:44.986845 13429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:44.986922 13429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:44.986971 13429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:44.998690 13429 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:37585
I20260925 02:52:44.998728 13558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:37585 every 8 connection(s)
I20260925 02:52:44.999434 13429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:52:45 dist-test-slave-252r krb5kdc[13186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304764, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:45.009130 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 13429
I20260925 02:52:45.009258 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:52:45.009549 13179 external_mini_cluster.cc:1428] Setting key efad9bd8226a801448988f67b96a5e73
I20260925 02:52:45.012312 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:45.001069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:59239 (local address 127.12.222.254:39931)
0925 02:52:45.001205 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:45.001210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:45.002248 (+ 1038us) server_negotiation.cc:407] Connection header received
0925 02:52:45.003204 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:45.003207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:45.003252 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:45.003329 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:45.004361 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:45.004876 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:45.006110 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:45.006317 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:45.006415 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:45.008824 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:45.008828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:45.008829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:45.008853 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:45.010297 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:45.010788 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:45.010791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:45.010793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:45.010874 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:45.011189 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:45.011192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:45.011194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:45.011402 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:45.011507 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:45.011917 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 02:52:45.012054 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:52:45.013149 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:45.001442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:39931 (local address 127.12.222.194:59239)
0925 02:52:45.002119 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:45.002158 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:45.003012 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:45.003441 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:45.003446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:45.003751 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:45.004179 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:45.004191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:45.004982 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:45.004984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:45.005998 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:45.006008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:45.006468 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:45.006471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:45.006594 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:45.007154 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:45.007177 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:45.008686 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:45.010391 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:45.010395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:45.010406 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:45.010690 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:45.010980 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:45.010982 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:45.010983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:45.011100 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:45.011505 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:45.011509 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:45.011691 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 02:52:45.011844 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 02:52:45.012057 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":108,"threads_started":1}
I20260925 02:52:45.014369 13559 heartbeater.cc:347] Connected to a master server at 127.12.222.254:39931
I20260925 02:52:45.014571 13559 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:45.014985 13559 heartbeater.cc:511] Master 127.12.222.254:39931 requested a full tablet report, sending...
I20260925 02:52:45.015978 13231 ts_manager.cc:194] Registered new tserver with Master: e1d3d1230a1746d4889c8e22fb49bbbc (127.12.222.194:37585)
I20260925 02:52:45.016886 13231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:59239
WARNING: no policy specified for kudu/127.12.222.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
I20260925 02:52:45.035244 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:39931
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:45.139678 13564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:45.139876 13564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:45.139912 13564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:45.143517 13564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:45.143576 13564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:45.143626 13564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:52:45.148068 13564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:39931
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:45.148847 13564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:45.149662 13564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:45.158425 13570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:45.158399 13569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:45.158403 13572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:45.158499 13564 server_base.cc:1034] running on GCE node
I20260925 02:52:45.158939 13564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:45.159427 13564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:45.160573 13564 hybrid_clock.cc:648] HybridClock initialized: now 1790304765160562 us; error 20 us; skew 500 ppm
Sep 25 02:52:45 dist-test-slave-252r krb5kdc[13186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304765, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:45.162925 13564 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:52:45.163635 13564 webserver.cc:460] Webserver started at http://127.12.222.195:37571/ using document root <none> and password file <none>
I20260925 02:52:45.164083 13564 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:45.164157 13564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:45.164376 13564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:45.166100 13564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "0f6d2163dbbd4a22a2ccf72018479acf"
format_stamp: "Formatted at 2026-09-25 02:52:45 on dist-test-slave-252r"
server_key: "8593bd8b87cb07630a5f4d6fb8b2fcff"
server_key_iv: "0ea5340380b48e40825ede3a89f8b292"
server_key_version: "encryptionkey@0"
I20260925 02:52:45.166577 13564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "0f6d2163dbbd4a22a2ccf72018479acf"
format_stamp: "Formatted at 2026-09-25 02:52:45 on dist-test-slave-252r"
server_key: "8593bd8b87cb07630a5f4d6fb8b2fcff"
server_key_iv: "0ea5340380b48e40825ede3a89f8b292"
server_key_version: "encryptionkey@0"
I20260925 02:52:45.169833 13564 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:52:45.171824 13579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:45.172942 13564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:45.173074 13564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "0f6d2163dbbd4a22a2ccf72018479acf"
format_stamp: "Formatted at 2026-09-25 02:52:45 on dist-test-slave-252r"
server_key: "8593bd8b87cb07630a5f4d6fb8b2fcff"
server_key_iv: "0ea5340380b48e40825ede3a89f8b292"
server_key_version: "encryptionkey@0"
I20260925 02:52:45.173200 13564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:45.207134 13564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:45.209704 13564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:45.209858 13564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:45.210412 13564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:52:45.211124 13564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:45.211182 13564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:45.211238 13564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:45.211279 13564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:45.220408 13564 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:39283
I20260925 02:52:45.220439 13692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:39283 every 8 connection(s)
I20260925 02:52:45.221197 13564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:52:45 dist-test-slave-252r krb5kdc[13186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304765, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:45.230054 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 13564
I20260925 02:52:45.230163 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:52:45.230437 13179 external_mini_cluster.cc:1428] Setting key afb997a1ade12d49207567459298d6d5
I20260925 02:52:45.233057 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:45.222817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:34281 (local address 127.12.222.254:39931)
0925 02:52:45.222985 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:45.222990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:45.223906 (+ 916us) server_negotiation.cc:407] Connection header received
0925 02:52:45.224733 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:45.224736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:45.224780 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:45.224859 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:45.225870 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:45.226365 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:45.227324 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:45.227506 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:45.227570 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:45.229668 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:45.229671 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:45.229673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:45.229692 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:45.231146 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:45.231624 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:45.231627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:45.231628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:45.231695 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:45.232058 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:45.232062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:45.232064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:45.232271 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:45.232362 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:45.232783 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 02:52:45.232887 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:52:45.233677 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:45.223178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:39931 (local address 127.12.222.195:34281)
0925 02:52:45.223778 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:45.223812 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:45.224546 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:45.225023 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:45.225028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:45.225340 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:45.225754 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:45.225767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:45.226465 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:45.226467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:45.227221 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:45.227240 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:45.227611 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:45.227613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:45.227728 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:45.228263 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:45.228286 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:45.229537 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:45.231250 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:45.231254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:45.231263 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:45.231535 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:45.231824 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:45.231826 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:45.231828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:45.231962 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:45.232361 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:45.232367 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:45.232568 (+ 201us) client_negotiation.cc:769] Sending connection context
0925 02:52:45.232737 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 02:52:45.233013 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":127,"threads_started":1}
I20260925 02:52:45.234525 13693 heartbeater.cc:347] Connected to a master server at 127.12.222.254:39931
I20260925 02:52:45.234712 13693 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:45.235092 13693 heartbeater.cc:511] Master 127.12.222.254:39931 requested a full tablet report, sending...
I20260925 02:52:45.235909 13231 ts_manager.cc:194] Registered new tserver with Master: 0f6d2163dbbd4a22a2ccf72018479acf (127.12.222.195:39283)
I20260925 02:52:45.236428 13231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:34281
I20260925 02:52:45.244297 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:52:45 dist-test-slave-252r krb5kdc[13186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304765, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:52:45 dist-test-slave-252r krb5kdc[13186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304765, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.222.193@KRBTEST.COM
I20260925 02:52:45.268358 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:45.259983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.12.222.193:35019)
0925 02:52:45.260283 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:45.260297 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:45.260319 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:52:45.260462 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:45.260466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:45.260606 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:45.260729 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:45.261393 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:45.262191 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:45.262986 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:45.263298 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:45.263404 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:45.265503 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:45.265511 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:45.265520 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:45.265541 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:45.266788 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:45.267172 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:45.267177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:45.267178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:45.267244 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:45.267479 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:45.267482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:45.267484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:45.267669 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:45.267857 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:45.268015 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 02:52:45.268160 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
W20260925 02:52:45.269121 13337 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56366
Sep 25 02:52:45 dist-test-slave-252r krb5kdc[13186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:52:45 dist-test-slave-252r krb5kdc[13186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.193@KRBTEST.COM
I20260925 02:52:45.290987 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:45.284132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56382 (local address 127.12.222.193:35019)
0925 02:52:45.284301 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:45.284305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:45.284371 (+ 66us) server_negotiation.cc:407] Connection header received
0925 02:52:45.284475 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:45.284477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:45.284510 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:45.284573 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:45.285172 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:45.285592 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:45.286277 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:45.286471 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:45.286534 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:45.288525 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:45.288528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:45.288529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:45.288566 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:45.289690 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:45.290036 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:45.290039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:45.290040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:45.290103 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:45.290312 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:45.290314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:45.290315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:45.290441 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:45.290542 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:45.290717 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 02:52:45.290797 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:52:45.291947 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 13293
I20260925 02:52:45.297902 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 13429
I20260925 02:52:45.303161 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 13564
I20260925 02:52:45.308295 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 13199
2026-09-25T02:52:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2382 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:45 dist-test-slave-252r krb5kdc[13708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:45 dist-test-slave-252r krb5kdc[13708](info): set up 2 sockets
Sep 25 02:52:45 dist-test-slave-252r krb5kdc[13708](info): commencing operation
krb5kdc: starting...
W20260925 02:52:46.272071 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:52:46 dist-test-slave-252r krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:52:46.381186 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:39049
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:46569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:39049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:46.482227 13721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:46.482451 13721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:46.482488 13721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:46.486191 13721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:46.486238 13721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:46.486256 13721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:46.486271 13721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:46.486286 13721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:46.490998 13721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:39049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:39049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:46.491930 13721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:46.492954 13721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:46.500890 13727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:46.500895 13726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:46.500993 13721 server_base.cc:1034] running on GCE node
W20260925 02:52:46.500898 13729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:46.501616 13721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:46.502573 13721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:46.503736 13721 hybrid_clock.cc:648] HybridClock initialized: now 1790304766503719 us; error 29 us; skew 500 ppm
Sep 25 02:52:46 dist-test-slave-252r krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304766, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:46.506292 13721 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:52:46.507031 13721 webserver.cc:460] Webserver started at http://127.12.222.254:41445/ using document root <none> and password file <none>
I20260925 02:52:46.507537 13721 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:46.507619 13721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:46.507825 13721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:46.509752 13721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "09042e3199764dd0a5fd3c00fd19913e"
format_stamp: "Formatted at 2026-09-25 02:52:46 on dist-test-slave-252r"
server_key: "92620955e98dbe6867fab49d062d88d6"
server_key_iv: "4ff7104d624c4af2daef355104dbc5eb"
server_key_version: "encryptionkey@0"
I20260925 02:52:46.510286 13721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "09042e3199764dd0a5fd3c00fd19913e"
format_stamp: "Formatted at 2026-09-25 02:52:46 on dist-test-slave-252r"
server_key: "92620955e98dbe6867fab49d062d88d6"
server_key_iv: "4ff7104d624c4af2daef355104dbc5eb"
server_key_version: "encryptionkey@0"
I20260925 02:52:46.513898 13721 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:52:46.517105 13736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:46.518523 13721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:52:46.518651 13721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "09042e3199764dd0a5fd3c00fd19913e"
format_stamp: "Formatted at 2026-09-25 02:52:46 on dist-test-slave-252r"
server_key: "92620955e98dbe6867fab49d062d88d6"
server_key_iv: "4ff7104d624c4af2daef355104dbc5eb"
server_key_version: "encryptionkey@0"
I20260925 02:52:46.518760 13721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:46.528415 13721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:46.530884 13721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:46.531026 13721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:46.537519 13721 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:39049
I20260925 02:52:46.537544 13788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:39049 every 8 connection(s)
I20260925 02:52:46.538338 13721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:46.540727 13789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:46.545641 13789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e: Bootstrap starting.
I20260925 02:52:46.546110 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 13721
I20260925 02:52:46.546216 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:52:46.546432 13179 external_mini_cluster.cc:1428] Setting key b848237fc3a794424dd09eb72c07a2fc
I20260925 02:52:46.548163 13789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:46.548890 13789 log.cc:824] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:46.550689 13789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e: No bootstrap required, opened a new log
Sep 25 02:52:46 dist-test-slave-252r krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:46.555065 13789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09042e3199764dd0a5fd3c00fd19913e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39049 } }
I20260925 02:52:46.555246 13789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:46.555307 13789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09042e3199764dd0a5fd3c00fd19913e, State: Initialized, Role: FOLLOWER
I20260925 02:52:46.555693 13789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09042e3199764dd0a5fd3c00fd19913e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39049 } }
I20260925 02:52:46.555817 13789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:46.555874 13789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:46.555984 13789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:46.556823 13789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09042e3199764dd0a5fd3c00fd19913e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39049 } }
I20260925 02:52:46.557134 13789 leader_election.cc:302] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09042e3199764dd0a5fd3c00fd19913e; no voters:
I20260925 02:52:46.557405 13789 leader_election.cc:288] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:46.557472 13794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:46.557790 13794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e [term 1 LEADER]: Becoming Leader. State: Replica: 09042e3199764dd0a5fd3c00fd19913e, State: Running, Role: LEADER
I20260925 02:52:46.558605 13794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09042e3199764dd0a5fd3c00fd19913e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39049 } }
I20260925 02:52:46.558868 13789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:46.559811 13794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09042e3199764dd0a5fd3c00fd19913e. Latest consensus state: current_term: 1 leader_uuid: "09042e3199764dd0a5fd3c00fd19913e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09042e3199764dd0a5fd3c00fd19913e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39049 } } }
I20260925 02:52:46.560199 13795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09042e3199764dd0a5fd3c00fd19913e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09042e3199764dd0a5fd3c00fd19913e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39049 } } }
I20260925 02:52:46.560336 13795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:46.560024 13794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:46.560976 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:46.547634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34210 (local address 127.12.222.254:39049)
0925 02:52:46.548059 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:46.548092 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:46.548124 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:52:46.548797 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:46.548813 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:46.549175 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:46.549489 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:46.550251 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:46.551388 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:46.552254 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:46.552706 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:46.552868 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:46.555227 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:46.555247 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:46.555263 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:46.555293 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:46.556980 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:46.557447 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:46.557453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:46.557458 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:46.557628 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:46.558008 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:46.558010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:46.558012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:46.558488 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:46.558654 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:46.559421 (+ 767us) server_negotiation.cc:299] Negotiation successful
0925 02:52:46.559714 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":114,"threads_started":1}
W20260925 02:52:46.564009 13809 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:52:46.564095 13809 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:52:46.564206 13804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:46.565027 13804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:46.569751 13804 catalog_manager.cc:1347] Generated new cluster ID: a22e3cc41b114afea3371962b22a29a8
I20260925 02:52:46.569808 13804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:46.584170 13804 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:46.586853 13804 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:46.593786 13804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09042e3199764dd0a5fd3c00fd19913e: Generated new TSK 0
I20260925 02:52:46.594314 13804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
I20260925 02:52:46.627233 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:39049
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:46569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:46.745121 13815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:46.745358 13815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:46.745426 13815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:46.748942 13815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:46.749010 13815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:46.749099 13815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:52:46.753906 13815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:39049
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:46.754773 13815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:46.755654 13815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:46.764556 13823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:46.764556 13820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:46.764590 13821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:46.764918 13815 server_base.cc:1034] running on GCE node
I20260925 02:52:46.765316 13815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:46.765889 13815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:46.767067 13815 hybrid_clock.cc:648] HybridClock initialized: now 1790304766767036 us; error 43 us; skew 500 ppm
Sep 25 02:52:46 dist-test-slave-252r krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304766, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:46.769667 13815 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:52:46.770360 13815 webserver.cc:460] Webserver started at http://127.12.222.193:33077/ using document root <none> and password file <none>
I20260925 02:52:46.770803 13815 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:46.770853 13815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:46.771028 13815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:46.772969 13815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "40ba301ebd1d4bb7a14354389a533052"
format_stamp: "Formatted at 2026-09-25 02:52:46 on dist-test-slave-252r"
server_key: "4bc9988be70ec700ad2acd79848506b5"
server_key_iv: "a91a26d284cbcfadf2d933dd9394c42b"
server_key_version: "encryptionkey@0"
I20260925 02:52:46.773464 13815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "40ba301ebd1d4bb7a14354389a533052"
format_stamp: "Formatted at 2026-09-25 02:52:46 on dist-test-slave-252r"
server_key: "4bc9988be70ec700ad2acd79848506b5"
server_key_iv: "a91a26d284cbcfadf2d933dd9394c42b"
server_key_version: "encryptionkey@0"
I20260925 02:52:46.776923 13815 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:52:46.778999 13830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:46.780097 13815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 02:52:46.780191 13815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "40ba301ebd1d4bb7a14354389a533052"
format_stamp: "Formatted at 2026-09-25 02:52:46 on dist-test-slave-252r"
server_key: "4bc9988be70ec700ad2acd79848506b5"
server_key_iv: "a91a26d284cbcfadf2d933dd9394c42b"
server_key_version: "encryptionkey@0"
I20260925 02:52:46.780319 13815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:46.803218 13815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:46.805598 13815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:46.805727 13815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:46.806164 13815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:52:46.806898 13815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:46.806947 13815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:46.806985 13815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:46.807006 13815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:46.814924 13815 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:44817
I20260925 02:52:46.814994 13944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:44817 every 8 connection(s)
I20260925 02:52:46.815779 13815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:52:46.822890 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 13815
I20260925 02:52:46.823009 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:52:46.823319 13179 external_mini_cluster.cc:1428] Setting key 61e3b2a1cd24ed2a8700e753aeaf2c9f
Sep 25 02:52:46 dist-test-slave-252r krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304766, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:46.829818 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:46.817525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:54811 (local address 127.12.222.254:39049)
0925 02:52:46.817653 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:46.817657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:46.818447 (+ 790us) server_negotiation.cc:407] Connection header received
0925 02:52:46.819737 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:46.819739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:46.819784 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:46.819856 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:46.821035 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:46.821570 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:46.822721 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:46.822945 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:46.823101 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:46.825942 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:46.825945 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:46.825947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:46.825967 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:46.827591 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:46.828050 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:46.828056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:46.828060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:46.828151 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:46.828538 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:46.828544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:46.828549 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:46.828785 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:46.828913 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:46.829446 (+ 533us) server_negotiation.cc:299] Negotiation successful
0925 02:52:46.829573 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:52:46.830323 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:46.817862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:39049 (local address 127.12.222.193:54811)
0925 02:52:46.818299 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:46.818340 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:46.819551 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:46.820014 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:46.820020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:46.820408 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:46.820850 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:46.820864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:46.821677 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:46.821679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:46.822586 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:46.822596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:46.823128 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:46.823134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:46.823325 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:46.823963 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:46.823978 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:46.825789 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:46.827702 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:46.827706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:46.827715 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:46.827951 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:46.828286 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:46.828293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:46.828295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:46.828424 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:46.828916 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:46.828924 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:46.829192 (+ 268us) client_negotiation.cc:769] Sending connection context
0925 02:52:46.829335 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 02:52:46.829558 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":109,"threads_started":1}
I20260925 02:52:46.831460 13945 heartbeater.cc:347] Connected to a master server at 127.12.222.254:39049
I20260925 02:52:46.831715 13945 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:46.832305 13945 heartbeater.cc:511] Master 127.12.222.254:39049 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.222.194@KRBTEST.COM; defaulting to no policy
I20260925 02:52:46.833792 13753 ts_manager.cc:194] Registered new tserver with Master: 40ba301ebd1d4bb7a14354389a533052 (127.12.222.193:44817)
I20260925 02:52:46.834820 13753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:54811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
I20260925 02:52:46.850380 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:39049
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:46569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:46.949672 13950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:46.949891 13950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:46.949934 13950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:46.953720 13950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:46.953763 13950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:46.953855 13950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:52:46.958451 13950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:39049
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:46.959314 13950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:46.960319 13950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:46.969063 13958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:46.969005 13955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:46.969004 13956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:46.969200 13950 server_base.cc:1034] running on GCE node
I20260925 02:52:46.969655 13950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:46.970155 13950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:46.971302 13950 hybrid_clock.cc:648] HybridClock initialized: now 1790304766971286 us; error 26 us; skew 500 ppm
Sep 25 02:52:46 dist-test-slave-252r krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304766, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:46.973758 13950 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:52:46.974471 13950 webserver.cc:460] Webserver started at http://127.12.222.194:39347/ using document root <none> and password file <none>
I20260925 02:52:46.974969 13950 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:46.975023 13950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:46.975209 13950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:46.977191 13950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "adc56ce34c6843d48f5d47c4875ffe45"
format_stamp: "Formatted at 2026-09-25 02:52:46 on dist-test-slave-252r"
server_key: "fc17e3867acea44b2234818241622721"
server_key_iv: "d77fea3c03fa6ea7d0f2588195581592"
server_key_version: "encryptionkey@0"
I20260925 02:52:46.977700 13950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "adc56ce34c6843d48f5d47c4875ffe45"
format_stamp: "Formatted at 2026-09-25 02:52:46 on dist-test-slave-252r"
server_key: "fc17e3867acea44b2234818241622721"
server_key_iv: "d77fea3c03fa6ea7d0f2588195581592"
server_key_version: "encryptionkey@0"
I20260925 02:52:46.981010 13950 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:52:46.983088 13965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:46.984160 13950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:46.984287 13950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "adc56ce34c6843d48f5d47c4875ffe45"
format_stamp: "Formatted at 2026-09-25 02:52:46 on dist-test-slave-252r"
server_key: "fc17e3867acea44b2234818241622721"
server_key_iv: "d77fea3c03fa6ea7d0f2588195581592"
server_key_version: "encryptionkey@0"
I20260925 02:52:46.984371 13950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:46.995862 13950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:46.998667 13950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:46.998806 13950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:46.999310 13950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:52:46.999992 13950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:47.000033 13950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:47.000088 13950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:47.000120 13950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:47.008038 13950 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:40503
I20260925 02:52:47.008082 14079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:40503 every 8 connection(s)
I20260925 02:52:47.008932 13950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:52:47.015661 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 13950
I20260925 02:52:47.015781 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:52:47.016124 13179 external_mini_cluster.cc:1428] Setting key d63dc9ac50e48e61081eaba86b480d0b
Sep 25 02:52:47 dist-test-slave-252r krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304766, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:47.021713 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:47.010443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:40573 (local address 127.12.222.254:39049)
0925 02:52:47.010556 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:47.010560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:47.011245 (+ 685us) server_negotiation.cc:407] Connection header received
0925 02:52:47.012471 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:47.012475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:47.012535 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:47.012615 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:47.013642 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.014160 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.015165 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.015375 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.015505 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:47.018039 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:47.018045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:47.018050 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:47.018079 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:47.019554 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:47.020100 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:47.020102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:47.020103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:47.020166 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:47.020551 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:47.020557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:47.020561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:47.020781 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:47.020907 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:47.021338 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 02:52:47.021486 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:52:47.022204 14082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:47.010700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:39049 (local address 127.12.222.194:40573)
0925 02:52:47.011112 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:47.011149 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:47.012220 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:47.012789 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:47.012795 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:47.013116 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:47.013511 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:47.013522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.014272 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.014276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:47.015049 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:47.015058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.015522 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.015526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:47.015696 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:47.016291 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:47.016307 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:47.017882 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:47.019658 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:47.019663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:47.019676 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:47.020005 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:47.020261 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:47.020266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:47.020268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:47.020443 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:47.020898 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:47.020903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:47.021111 (+ 208us) client_negotiation.cc:769] Sending connection context
0925 02:52:47.021242 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 02:52:47.021439 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":124,"threads_started":1}
I20260925 02:52:47.023265 14080 heartbeater.cc:347] Connected to a master server at 127.12.222.254:39049
I20260925 02:52:47.023495 14080 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:47.023897 14080 heartbeater.cc:511] Master 127.12.222.254:39049 requested a full tablet report, sending...
I20260925 02:52:47.024887 13753 ts_manager.cc:194] Registered new tserver with Master: adc56ce34c6843d48f5d47c4875ffe45 (127.12.222.194:40503)
I20260925 02:52:47.025427 13753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:40573
WARNING: no policy specified for kudu/127.12.222.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
I20260925 02:52:47.040776 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:39049
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:46569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:47.142978 14085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:47.143219 14085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:47.143261 14085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:47.147267 14085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:47.147318 14085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:47.147423 14085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:52:47.152523 14085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:39049
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:47.153477 14085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:47.154428 14085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:47.163275 14090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:47.163336 14093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:47.163425 14085 server_base.cc:1034] running on GCE node
W20260925 02:52:47.163275 14091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:47.163885 14085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:47.164497 14085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:47.165645 14085 hybrid_clock.cc:648] HybridClock initialized: now 1790304767165633 us; error 23 us; skew 500 ppm
Sep 25 02:52:47 dist-test-slave-252r krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304767, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:47.168155 14085 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:52:47.168931 14085 webserver.cc:460] Webserver started at http://127.12.222.195:46131/ using document root <none> and password file <none>
I20260925 02:52:47.169418 14085 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:47.169474 14085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:47.169651 14085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:47.171499 14085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "604fca526e804a7c9081b8c0deac4b3c"
format_stamp: "Formatted at 2026-09-25 02:52:47 on dist-test-slave-252r"
server_key: "8e40808bf0b32ab4e446121ceeef3be7"
server_key_iv: "f791b5115b2f495772c67412a3dae47c"
server_key_version: "encryptionkey@0"
I20260925 02:52:47.172075 14085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "604fca526e804a7c9081b8c0deac4b3c"
format_stamp: "Formatted at 2026-09-25 02:52:47 on dist-test-slave-252r"
server_key: "8e40808bf0b32ab4e446121ceeef3be7"
server_key_iv: "f791b5115b2f495772c67412a3dae47c"
server_key_version: "encryptionkey@0"
I20260925 02:52:47.175437 14085 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:52:47.177603 14100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:47.178639 14085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:52:47.178762 14085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "604fca526e804a7c9081b8c0deac4b3c"
format_stamp: "Formatted at 2026-09-25 02:52:47 on dist-test-slave-252r"
server_key: "8e40808bf0b32ab4e446121ceeef3be7"
server_key_iv: "f791b5115b2f495772c67412a3dae47c"
server_key_version: "encryptionkey@0"
I20260925 02:52:47.178861 14085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:47.205797 14085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:47.208277 14085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:47.208453 14085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:47.208974 14085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:52:47.209699 14085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:47.209761 14085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:47.209832 14085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:47.209872 14085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:47.218662 14085 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:46387
I20260925 02:52:47.218694 14214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:46387 every 8 connection(s)
I20260925 02:52:47.219492 14085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:52:47.225797 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 14085
I20260925 02:52:47.225895 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:52:47.226173 13179 external_mini_cluster.cc:1428] Setting key a46aaaa1da99009ece6c3836c4c511cd
Sep 25 02:52:47 dist-test-slave-252r krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304767, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:47.231844 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:47.221162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:49147 (local address 127.12.222.254:39049)
0925 02:52:47.221302 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:47.221306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:47.222053 (+ 747us) server_negotiation.cc:407] Connection header received
0925 02:52:47.223247 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:47.223250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:47.223307 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:47.223389 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:47.224526 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.225029 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.226101 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.226397 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.226503 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:47.229050 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:47.229053 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:47.229055 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:47.229076 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:47.230219 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:47.230681 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:47.230684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:47.230685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:47.230752 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:47.231031 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:47.231033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:47.231034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:47.231183 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:47.231255 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:47.231650 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 02:52:47.231734 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:52:47.232673 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:47.221501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:39049 (local address 127.12.222.195:49147)
0925 02:52:47.221890 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:47.221954 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:47.223020 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:47.223556 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:47.223562 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:47.223871 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:47.224394 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:47.224411 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.225130 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.225132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:47.225990 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:47.225999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.226559 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.226565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:47.226733 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:47.227302 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:47.227321 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:47.228905 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:47.230350 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:47.230354 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:47.230364 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:47.230600 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:47.230826 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:47.230829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:47.230830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:47.230958 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:47.231264 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:47.231268 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:47.231467 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 02:52:47.231608 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 02:52:47.231819 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":96,"threads_started":1}
I20260925 02:52:47.233681 14215 heartbeater.cc:347] Connected to a master server at 127.12.222.254:39049
I20260925 02:52:47.233875 14215 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:47.234287 14215 heartbeater.cc:511] Master 127.12.222.254:39049 requested a full tablet report, sending...
I20260925 02:52:47.235169 13753 ts_manager.cc:194] Registered new tserver with Master: 604fca526e804a7c9081b8c0deac4b3c (127.12.222.195:46387)
I20260925 02:52:47.235623 13753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:49147
I20260925 02:52:47.239723 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:52:47.249800 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:47.243651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34214 (local address 127.12.222.254:39049)
0925 02:52:47.243792 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:47.243795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:47.243987 (+ 192us) server_negotiation.cc:407] Connection header received
0925 02:52:47.244069 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:47.244071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:47.244103 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:47.244167 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:47.244853 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.245322 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.246033 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.246222 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.246289 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:47.247436 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:47.247439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:47.247441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:47.247460 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:47.248617 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:47.248945 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:47.248954 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:47.248955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:47.249021 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:47.249225 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:47.249227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:47.249228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:47.249397 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:47.249489 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:47.249626 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 02:52:47.249704 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:52:47.254273 13753 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:52:47.257272 13753 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:52:47.270470 14226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:47.264966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:46387 (local address 127.0.0.1:48264)
0925 02:52:47.265253 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:47.265293 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:47.265401 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:47.266585 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:47.266591 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:47.266612 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:47.266749 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:47.266756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.267680 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.267685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:47.268760 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:47.268770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.269920 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.269924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:47.270035 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:47.270074 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 02:52:47.270172 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 02:52:47.270288 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260925 02:52:47.270660 14228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:47.265319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:40503 (local address 127.0.0.1:57708)
0925 02:52:47.265630 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:47.265645 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:47.265735 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:47.266655 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:47.266658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:47.266675 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:47.266768 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:47.266777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.268253 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.268257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:47.269256 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:47.269265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.270188 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.270191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:47.270282 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:47.270307 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:52:47.270382 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:52:47.270496 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":29,"thread_start_us":92,"threads_started":1}
I20260925 02:52:47.270838 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:47.265062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48264 (local address 127.12.222.195:46387)
0925 02:52:47.265955 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:47.265970 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:47.265993 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:52:47.266066 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:47.266073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:47.266261 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:47.266406 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:47.266848 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.267554 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.269072 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.269778 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.269910 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:47.270033 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:47.270384 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 02:52:47.270618 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":98,"threads_started":1}
I20260925 02:52:47.270853 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:47.265618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:44817 (local address 127.0.0.1:58324)
0925 02:52:47.265923 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:47.265947 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:47.266029 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:47.266724 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:47.266726 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:47.266740 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:47.266902 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:47.266910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.268455 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.268458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:47.269459 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:47.269469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.270479 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.270482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:47.270580 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:47.270599 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 02:52:47.270649 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 02:52:47.270721 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":38,"thread_start_us":94,"threads_started":1}
I20260925 02:52:47.271078 14229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:47.265539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57708 (local address 127.12.222.194:40503)
0925 02:52:47.266203 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:47.266217 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:47.266239 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:52:47.266313 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:47.266318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:47.266524 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:47.266660 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:47.266919 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.268099 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.269379 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.270051 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.270196 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:47.270303 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:47.270844 (+ 541us) server_negotiation.cc:299] Negotiation successful
0925 02:52:47.270946 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":121,"threads_started":1}
I20260925 02:52:47.271697 14231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:47.265774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.12.222.193:44817)
0925 02:52:47.266272 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:47.266284 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:47.266304 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:52:47.266380 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:47.266385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:47.266566 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:47.267059 (+ 493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:47.267275 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.268325 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.269575 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.270322 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.271358 (+ 1036us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:47.271441 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:47.271487 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 02:52:47.271568 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":110,"threads_started":1}
I20260925 02:52:47.273623 14013 tablet_service.cc:1467] Processing CreateTablet for tablet d8291b011d434d69a9dbc219825e44a7 (DEFAULT_TABLE table=test-table [id=66eb4ab7d1ec4f19a7d38f6f1efd7e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:47.273733 14148 tablet_service.cc:1467] Processing CreateTablet for tablet d8291b011d434d69a9dbc219825e44a7 (DEFAULT_TABLE table=test-table [id=66eb4ab7d1ec4f19a7d38f6f1efd7e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:47.274154 13879 tablet_service.cc:1467] Processing CreateTablet for tablet d8291b011d434d69a9dbc219825e44a7 (DEFAULT_TABLE table=test-table [id=66eb4ab7d1ec4f19a7d38f6f1efd7e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:47.274683 14013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8291b011d434d69a9dbc219825e44a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:47.274727 14148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8291b011d434d69a9dbc219825e44a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:47.275009 13879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8291b011d434d69a9dbc219825e44a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:47.281528 14232 tablet_bootstrap.cc:492] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45: Bootstrap starting.
I20260925 02:52:47.282136 14234 tablet_bootstrap.cc:492] T d8291b011d434d69a9dbc219825e44a7 P 604fca526e804a7c9081b8c0deac4b3c: Bootstrap starting.
I20260925 02:52:47.282399 14233 tablet_bootstrap.cc:492] T d8291b011d434d69a9dbc219825e44a7 P 40ba301ebd1d4bb7a14354389a533052: Bootstrap starting.
I20260925 02:52:47.283944 14232 tablet_bootstrap.cc:654] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:47.284482 14233 tablet_bootstrap.cc:654] T d8291b011d434d69a9dbc219825e44a7 P 40ba301ebd1d4bb7a14354389a533052: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:47.284479 14234 tablet_bootstrap.cc:654] T d8291b011d434d69a9dbc219825e44a7 P 604fca526e804a7c9081b8c0deac4b3c: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:47.284686 14232 log.cc:824] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:47.285760 14234 log.cc:824] T d8291b011d434d69a9dbc219825e44a7 P 604fca526e804a7c9081b8c0deac4b3c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:47.285760 14233 log.cc:824] T d8291b011d434d69a9dbc219825e44a7 P 40ba301ebd1d4bb7a14354389a533052: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:47.286495 14232 tablet_bootstrap.cc:492] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45: No bootstrap required, opened a new log
I20260925 02:52:47.286706 14232 ts_tablet_manager.cc:1397] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:52:47.287463 14234 tablet_bootstrap.cc:492] T d8291b011d434d69a9dbc219825e44a7 P 604fca526e804a7c9081b8c0deac4b3c: No bootstrap required, opened a new log
I20260925 02:52:47.287631 14234 ts_tablet_manager.cc:1397] T d8291b011d434d69a9dbc219825e44a7 P 604fca526e804a7c9081b8c0deac4b3c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:52:47.289424 14232 raft_consensus.cc:348] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604fca526e804a7c9081b8c0deac4b3c" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46387 } } peers { permanent_uuid: "40ba301ebd1d4bb7a14354389a533052" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44817 } } peers { permanent_uuid: "adc56ce34c6843d48f5d47c4875ffe45" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40503 } }
I20260925 02:52:47.289650 14232 raft_consensus.cc:374] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:47.289665 14233 tablet_bootstrap.cc:492] T d8291b011d434d69a9dbc219825e44a7 P 40ba301ebd1d4bb7a14354389a533052: No bootstrap required, opened a new log
I20260925 02:52:47.289706 14232 raft_consensus.cc:729] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc56ce34c6843d48f5d47c4875ffe45, State: Initialized, Role: FOLLOWER
I20260925 02:52:47.289808 14233 ts_tablet_manager.cc:1397] T d8291b011d434d69a9dbc219825e44a7 P 40ba301ebd1d4bb7a14354389a533052: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260925 02:52:47.290115 14232 consensus_queue.cc:260] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604fca526e804a7c9081b8c0deac4b3c" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46387 } } peers { permanent_uuid: "40ba301ebd1d4bb7a14354389a533052" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44817 } } peers { permanent_uuid: "adc56ce34c6843d48f5d47c4875ffe45" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40503 } }
I20260925 02:52:47.290693 14080 heartbeater.cc:503] Master 127.12.222.254:39049 was elected leader, sending a full tablet report...
I20260925 02:52:47.291038 14232 ts_tablet_manager.cc:1428] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260925 02:52:47.291072 14234 raft_consensus.cc:348] T d8291b011d434d69a9dbc219825e44a7 P 604fca526e804a7c9081b8c0deac4b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604fca526e804a7c9081b8c0deac4b3c" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46387 } } peers { permanent_uuid: "40ba301ebd1d4bb7a14354389a533052" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44817 } } peers { permanent_uuid: "adc56ce34c6843d48f5d47c4875ffe45" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40503 } }
I20260925 02:52:47.291324 14234 raft_consensus.cc:374] T d8291b011d434d69a9dbc219825e44a7 P 604fca526e804a7c9081b8c0deac4b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:47.291383 14234 raft_consensus.cc:729] T d8291b011d434d69a9dbc219825e44a7 P 604fca526e804a7c9081b8c0deac4b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604fca526e804a7c9081b8c0deac4b3c, State: Initialized, Role: FOLLOWER
I20260925 02:52:47.291806 14234 consensus_queue.cc:260] T d8291b011d434d69a9dbc219825e44a7 P 604fca526e804a7c9081b8c0deac4b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604fca526e804a7c9081b8c0deac4b3c" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46387 } } peers { permanent_uuid: "40ba301ebd1d4bb7a14354389a533052" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44817 } } peers { permanent_uuid: "adc56ce34c6843d48f5d47c4875ffe45" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40503 } }
I20260925 02:52:47.292479 14215 heartbeater.cc:503] Master 127.12.222.254:39049 was elected leader, sending a full tablet report...
I20260925 02:52:47.292802 14234 ts_tablet_manager.cc:1428] T d8291b011d434d69a9dbc219825e44a7 P 604fca526e804a7c9081b8c0deac4b3c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 02:52:47.293433 14233 raft_consensus.cc:348] T d8291b011d434d69a9dbc219825e44a7 P 40ba301ebd1d4bb7a14354389a533052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604fca526e804a7c9081b8c0deac4b3c" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46387 } } peers { permanent_uuid: "40ba301ebd1d4bb7a14354389a533052" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44817 } } peers { permanent_uuid: "adc56ce34c6843d48f5d47c4875ffe45" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40503 } }
I20260925 02:52:47.293685 14233 raft_consensus.cc:374] T d8291b011d434d69a9dbc219825e44a7 P 40ba301ebd1d4bb7a14354389a533052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:47.293746 14233 raft_consensus.cc:729] T d8291b011d434d69a9dbc219825e44a7 P 40ba301ebd1d4bb7a14354389a533052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ba301ebd1d4bb7a14354389a533052, State: Initialized, Role: FOLLOWER
I20260925 02:52:47.294152 14233 consensus_queue.cc:260] T d8291b011d434d69a9dbc219825e44a7 P 40ba301ebd1d4bb7a14354389a533052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604fca526e804a7c9081b8c0deac4b3c" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46387 } } peers { permanent_uuid: "40ba301ebd1d4bb7a14354389a533052" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44817 } } peers { permanent_uuid: "adc56ce34c6843d48f5d47c4875ffe45" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40503 } }
I20260925 02:52:47.294898 14233 ts_tablet_manager.cc:1428] T d8291b011d434d69a9dbc219825e44a7 P 40ba301ebd1d4bb7a14354389a533052: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 02:52:47.295213 13945 heartbeater.cc:503] Master 127.12.222.254:39049 was elected leader, sending a full tablet report...
W20260925 02:52:47.316891 13946 tablet.cc:2369] T d8291b011d434d69a9dbc219825e44a7 P 40ba301ebd1d4bb7a14354389a533052: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:52:47.470376 14216 tablet.cc:2369] T d8291b011d434d69a9dbc219825e44a7 P 604fca526e804a7c9081b8c0deac4b3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:52:47.509907 14081 tablet.cc:2369] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:52:47.525936 14238 raft_consensus.cc:482] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:52:47.526137 14238 raft_consensus.cc:504] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604fca526e804a7c9081b8c0deac4b3c" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46387 } } peers { permanent_uuid: "40ba301ebd1d4bb7a14354389a533052" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44817 } } peers { permanent_uuid: "adc56ce34c6843d48f5d47c4875ffe45" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40503 } }
I20260925 02:52:47.527555 14238 leader_election.cc:288] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 604fca526e804a7c9081b8c0deac4b3c (127.12.222.195:46387), 40ba301ebd1d4bb7a14354389a533052 (127.12.222.193:44817)
I20260925 02:52:47.532717 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:47.527869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:44781 (local address 127.12.222.195:46387)
0925 02:52:47.528005 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:47.528012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:47.528562 (+ 550us) server_negotiation.cc:407] Connection header received
0925 02:52:47.528694 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:47.528700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:47.528756 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:47.528826 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:47.529277 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.529783 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.531384 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.532131 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.532268 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:47.532321 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:47.532532 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 02:52:47.532595 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:52:47.532732 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:47.527719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:46387 (local address 127.12.222.194:44781)
0925 02:52:47.528442 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:47.528467 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:47.528599 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:47.528853 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:47.528856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:47.528874 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:47.529031 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:47.529037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.529891 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.529896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:47.531182 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:47.531193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.532273 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.532276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:47.532372 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:47.532394 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:52:47.532518 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 02:52:47.532587 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":81,"threads_started":1}
I20260925 02:52:47.533485 14168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8291b011d434d69a9dbc219825e44a7" candidate_uuid: "adc56ce34c6843d48f5d47c4875ffe45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604fca526e804a7c9081b8c0deac4b3c" is_pre_election: true
I20260925 02:52:47.533953 14168 raft_consensus.cc:2454] T d8291b011d434d69a9dbc219825e44a7 P 604fca526e804a7c9081b8c0deac4b3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adc56ce34c6843d48f5d47c4875ffe45 in term 0.
I20260925 02:52:47.534188 14231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:47.528292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:46909 (local address 127.12.222.193:44817)
0925 02:52:47.528420 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:47.528425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:47.528783 (+ 358us) server_negotiation.cc:407] Connection header received
0925 02:52:47.529032 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:47.529035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:47.529098 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:47.529175 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:47.529590 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.530640 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.532875 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.533492 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.533586 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:47.533632 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:47.534001 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 02:52:47.534080 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:52:47.534245 14242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:47.528112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:44817 (local address 127.12.222.194:46909)
0925 02:52:47.528669 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:47.528689 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:47.528867 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:47.529207 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:47.529213 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:47.529236 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:47.529429 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:47.529438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.530817 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.530822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:47.532706 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:47.532721 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.533725 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.533730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:47.533842 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:47.533866 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:52:47.533962 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 02:52:47.534071 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":64,"threads_started":1}
I20260925 02:52:47.534583 13969 leader_election.cc:302] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 604fca526e804a7c9081b8c0deac4b3c, adc56ce34c6843d48f5d47c4875ffe45; no voters:
I20260925 02:52:47.534814 13899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8291b011d434d69a9dbc219825e44a7" candidate_uuid: "adc56ce34c6843d48f5d47c4875ffe45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40ba301ebd1d4bb7a14354389a533052" is_pre_election: true
I20260925 02:52:47.534875 14238 raft_consensus.cc:2781] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:52:47.535018 14238 raft_consensus.cc:482] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:52:47.535090 14238 raft_consensus.cc:3037] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:47.535108 13899 raft_consensus.cc:2454] T d8291b011d434d69a9dbc219825e44a7 P 40ba301ebd1d4bb7a14354389a533052 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adc56ce34c6843d48f5d47c4875ffe45 in term 0.
I20260925 02:52:47.536334 14238 raft_consensus.cc:504] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604fca526e804a7c9081b8c0deac4b3c" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46387 } } peers { permanent_uuid: "40ba301ebd1d4bb7a14354389a533052" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44817 } } peers { permanent_uuid: "adc56ce34c6843d48f5d47c4875ffe45" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40503 } }
I20260925 02:52:47.536671 14238 leader_election.cc:288] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45 [CANDIDATE]: Term 1 election: Requested vote from peers 604fca526e804a7c9081b8c0deac4b3c (127.12.222.195:46387), 40ba301ebd1d4bb7a14354389a533052 (127.12.222.193:44817)
I20260925 02:52:47.537074 14168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8291b011d434d69a9dbc219825e44a7" candidate_uuid: "adc56ce34c6843d48f5d47c4875ffe45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604fca526e804a7c9081b8c0deac4b3c"
I20260925 02:52:47.537211 14168 raft_consensus.cc:3037] T d8291b011d434d69a9dbc219825e44a7 P 604fca526e804a7c9081b8c0deac4b3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:47.537452 13899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8291b011d434d69a9dbc219825e44a7" candidate_uuid: "adc56ce34c6843d48f5d47c4875ffe45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40ba301ebd1d4bb7a14354389a533052"
I20260925 02:52:47.537576 13899 raft_consensus.cc:3037] T d8291b011d434d69a9dbc219825e44a7 P 40ba301ebd1d4bb7a14354389a533052 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:47.538513 14168 raft_consensus.cc:2454] T d8291b011d434d69a9dbc219825e44a7 P 604fca526e804a7c9081b8c0deac4b3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc56ce34c6843d48f5d47c4875ffe45 in term 1.
I20260925 02:52:47.538803 13899 raft_consensus.cc:2454] T d8291b011d434d69a9dbc219825e44a7 P 40ba301ebd1d4bb7a14354389a533052 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc56ce34c6843d48f5d47c4875ffe45 in term 1.
I20260925 02:52:47.538898 13969 leader_election.cc:302] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 604fca526e804a7c9081b8c0deac4b3c, adc56ce34c6843d48f5d47c4875ffe45; no voters:
I20260925 02:52:47.539173 14238 raft_consensus.cc:2781] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:47.539372 14238 raft_consensus.cc:686] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45 [term 1 LEADER]: Becoming Leader. State: Replica: adc56ce34c6843d48f5d47c4875ffe45, State: Running, Role: LEADER
I20260925 02:52:47.539913 14238 consensus_queue.cc:237] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604fca526e804a7c9081b8c0deac4b3c" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46387 } } peers { permanent_uuid: "40ba301ebd1d4bb7a14354389a533052" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44817 } } peers { permanent_uuid: "adc56ce34c6843d48f5d47c4875ffe45" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40503 } }
I20260925 02:52:47.543759 13752 catalog_manager.cc:5495] T d8291b011d434d69a9dbc219825e44a7 P adc56ce34c6843d48f5d47c4875ffe45 reported cstate change: term changed from 0 to 1, leader changed from <none> to adc56ce34c6843d48f5d47c4875ffe45 (127.12.222.194). New cstate: current_term: 1 leader_uuid: "adc56ce34c6843d48f5d47c4875ffe45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604fca526e804a7c9081b8c0deac4b3c" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40ba301ebd1d4bb7a14354389a533052" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adc56ce34c6843d48f5d47c4875ffe45" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40503 } health_report { overall_health: HEALTHY } } }
Sep 25 02:52:47 dist-test-slave-252r krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.193@KRBTEST.COM
I20260925 02:52:47.562125 14231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:47.552230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58328 (local address 127.12.222.193:44817)
0925 02:52:47.552375 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:47.552379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:47.552580 (+ 201us) server_negotiation.cc:407] Connection header received
0925 02:52:47.552700 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:47.552705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:47.552762 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:47.552843 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:47.553593 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.554368 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.555550 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.555940 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.556068 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:47.558416 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:47.558425 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:47.558434 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:47.558455 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:47.560159 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:47.560668 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:47.560674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:47.560676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:47.560781 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:47.561122 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:47.561125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:47.561126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:47.561412 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:47.561595 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:47.561838 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 02:52:47.561968 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 02:52:47 dist-test-slave-252r krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304767, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:52:47 dist-test-slave-252r krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304767, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.222.193@KRBTEST.COM
I20260925 02:52:47.586635 14231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:47.579319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58344 (local address 127.12.222.193:44817)
0925 02:52:47.579436 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:47.579440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:47.579560 (+ 120us) server_negotiation.cc:407] Connection header received
0925 02:52:47.579690 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:47.579696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:47.579746 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:47.579816 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:47.580421 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.580899 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.581695 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.581937 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.582040 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:47.584347 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:47.584350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:47.584352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:47.584386 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:47.585442 (+ 1056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:47.585779 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:47.585781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:47.585782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:47.585845 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:47.586042 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:47.586044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:47.586045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:47.586170 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:47.586255 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:47.586457 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 02:52:47.586536 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 02:52:47 dist-test-slave-252r krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:52:47 dist-test-slave-252r krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.193@KRBTEST.COM
I20260925 02:52:47.609462 14231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:47.602603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.12.222.193:44817)
0925 02:52:47.602733 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:47.602738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:47.602829 (+ 91us) server_negotiation.cc:407] Connection header received
0925 02:52:47.602922 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:47.602924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:47.602962 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:47.603026 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:47.603580 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.604028 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.604805 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:47.604987 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:47.605056 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:47.607053 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:47.607056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:47.607057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:47.607075 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:47.608122 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:47.608528 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:47.608530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:47.608531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:47.608599 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:47.608818 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:47.608820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:47.608821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:47.608955 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:47.609044 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:47.609208 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 02:52:47.609284 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 02:52:47.610733 13858 tablet_service.cc:2908] Rejecting scan request for tablet d8291b011d434d69a9dbc219825e44a7: Uninitialized: clean time has not yet been initialized
I20260925 02:52:47.612782 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 13815
I20260925 02:52:47.619609 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 13950
I20260925 02:52:47.625111 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 14085
I20260925 02:52:47.630149 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 13721
2026-09-25T02:52:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2322 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:47 dist-test-slave-252r krb5kdc[14257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:47 dist-test-slave-252r krb5kdc[14257](info): set up 2 sockets
Sep 25 02:52:47 dist-test-slave-252r krb5kdc[14257](info): commencing operation
krb5kdc: starting...
W20260925 02:52:48.599565 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:52:48 dist-test-slave-252r krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:52:48.705267 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:33123
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:33083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:33123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:48.803892 14270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:48.804148 14270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:48.804219 14270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:48.808084 14270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:48.808127 14270 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:52:48.808142 14270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:48.808195 14270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:48.808238 14270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:48.808264 14270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:48.812927 14270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:33083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:33123
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:33123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:48.813820 14270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:48.814666 14270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:48.822072 14276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:48.822041 14275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:48.822158 14270 server_base.cc:1034] running on GCE node
W20260925 02:52:48.822041 14278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:48.822858 14270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:48.823788 14270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:48.824937 14270 hybrid_clock.cc:648] HybridClock initialized: now 1790304768824919 us; error 30 us; skew 500 ppm
Sep 25 02:52:48 dist-test-slave-252r krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304768, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:48.831653 14270 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:52:48.832490 14270 webserver.cc:460] Webserver started at http://127.12.222.254:33637/ using document root <none> and password file <none>
I20260925 02:52:48.832973 14270 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:48.833030 14270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:48.833233 14270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:48.834976 14270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "505a116f63b24500a4b5eeb9fe873447"
format_stamp: "Formatted at 2026-09-25 02:52:48 on dist-test-slave-252r"
server_key: "c506dfa6595e73905e3e681ba2c76b82"
server_key_iv: "7130d354edb9898318284fcfabf01d8c"
server_key_version: "encryptionkey@0"
I20260925 02:52:48.835491 14270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "505a116f63b24500a4b5eeb9fe873447"
format_stamp: "Formatted at 2026-09-25 02:52:48 on dist-test-slave-252r"
server_key: "c506dfa6595e73905e3e681ba2c76b82"
server_key_iv: "7130d354edb9898318284fcfabf01d8c"
server_key_version: "encryptionkey@0"
I20260925 02:52:48.838758 14270 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 02:52:48.840864 14285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:48.841924 14270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:48.842052 14270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "505a116f63b24500a4b5eeb9fe873447"
format_stamp: "Formatted at 2026-09-25 02:52:48 on dist-test-slave-252r"
server_key: "c506dfa6595e73905e3e681ba2c76b82"
server_key_iv: "7130d354edb9898318284fcfabf01d8c"
server_key_version: "encryptionkey@0"
I20260925 02:52:48.842149 14270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:48.852993 14270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:48.855195 14270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:48.855360 14270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:48.862187 14270 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:33123
I20260925 02:52:48.862214 14348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:33123 every 8 connection(s)
I20260925 02:52:48.862952 14270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:48.865192 14349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:48.869403 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 14270
I20260925 02:52:48.869501 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:52:48.869868 13179 external_mini_cluster.cc:1428] Setting key ef2cf58c737459ba7414423188ed41a8
I20260925 02:52:48.869889 14349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447: Bootstrap starting.
I20260925 02:52:48.872131 14349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:48.872841 14349 log.cc:824] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:48.874456 14349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447: No bootstrap required, opened a new log
Sep 25 02:52:48 dist-test-slave-252r krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:48.879566 14349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505a116f63b24500a4b5eeb9fe873447" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33123 } }
I20260925 02:52:48.879806 14349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:48.879871 14349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 505a116f63b24500a4b5eeb9fe873447, State: Initialized, Role: FOLLOWER
I20260925 02:52:48.880481 14349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505a116f63b24500a4b5eeb9fe873447" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33123 } }
I20260925 02:52:48.880596 14349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:48.880671 14349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:48.880774 14349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:48.881676 14349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505a116f63b24500a4b5eeb9fe873447" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33123 } }
I20260925 02:52:48.882027 14349 leader_election.cc:302] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 505a116f63b24500a4b5eeb9fe873447; no voters:
I20260925 02:52:48.882335 14349 leader_election.cc:288] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:48.882443 14355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:48.882685 14355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447 [term 1 LEADER]: Becoming Leader. State: Replica: 505a116f63b24500a4b5eeb9fe873447, State: Running, Role: LEADER
I20260925 02:52:48.882943 14355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505a116f63b24500a4b5eeb9fe873447" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33123 } }
I20260925 02:52:48.883244 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:48.871060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.12.222.254:33123)
0925 02:52:48.871494 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:48.871508 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:48.871539 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:52:48.872141 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:48.872157 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:48.872532 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:48.872830 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:48.873581 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:48.874479 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:48.875614 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:48.876053 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:48.876168 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:48.878416 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:48.878422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:48.878437 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:48.878474 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:48.880293 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:48.880705 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:48.880712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:48.880719 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:48.880899 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:48.881176 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:48.881178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:48.881179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:48.881502 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:48.881638 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:48.881994 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 02:52:48.882268 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":164,"threads_started":1}
I20260925 02:52:48.883515 14349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:48.883981 14355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 505a116f63b24500a4b5eeb9fe873447. Latest consensus state: current_term: 1 leader_uuid: "505a116f63b24500a4b5eeb9fe873447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505a116f63b24500a4b5eeb9fe873447" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33123 } } }
I20260925 02:52:48.884084 14355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:48.884399 14357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "505a116f63b24500a4b5eeb9fe873447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505a116f63b24500a4b5eeb9fe873447" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33123 } } }
I20260925 02:52:48.884522 14357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447 [sys.catalog]: This master's current role is: LEADER
W20260925 02:52:48.888037 14370 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:52:48.888115 14370 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:52:48.888185 14360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:48.888785 14360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:48.893150 14360 catalog_manager.cc:1347] Generated new cluster ID: f441b708498147e2a46f9271837f09e9
I20260925 02:52:48.893208 14360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:48.898826 14360 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:48.899555 14360 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:48.907171 14360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447: Generated new TSK 0
I20260925 02:52:48.907652 14360 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
I20260925 02:52:48.942799 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:33123
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:33083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:49.063650 14376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:49.063881 14376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:49.063920 14376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:49.067670 14376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:49.067713 14376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:49.067747 14376 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:52:49.067781 14376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:52:49.072404 14376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:33083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.222.254:33123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:49.073201 14376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:49.074013 14376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:49.082602 14381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:49.082608 14382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:49.082770 14376 server_base.cc:1034] running on GCE node
W20260925 02:52:49.082608 14384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:49.083317 14376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:49.083827 14376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:49.084976 14376 hybrid_clock.cc:648] HybridClock initialized: now 1790304769084956 us; error 31 us; skew 500 ppm
Sep 25 02:52:49 dist-test-slave-252r krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304769, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:49.087468 14376 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:52:49.088263 14376 webserver.cc:460] Webserver started at http://127.12.222.193:41409/ using document root <none> and password file <none>
I20260925 02:52:49.088766 14376 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:49.088847 14376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:49.089057 14376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:49.090905 14376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "edfff45819664869907c7eeeccdd3ac5"
format_stamp: "Formatted at 2026-09-25 02:52:49 on dist-test-slave-252r"
server_key: "601d9e8178a06f2e1e695b4f6cfbf294"
server_key_iv: "3787f0bab7d6f2001c4fa2dfda1ab8c4"
server_key_version: "encryptionkey@0"
I20260925 02:52:49.091446 14376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "edfff45819664869907c7eeeccdd3ac5"
format_stamp: "Formatted at 2026-09-25 02:52:49 on dist-test-slave-252r"
server_key: "601d9e8178a06f2e1e695b4f6cfbf294"
server_key_iv: "3787f0bab7d6f2001c4fa2dfda1ab8c4"
server_key_version: "encryptionkey@0"
I20260925 02:52:49.094830 14376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 02:52:49.096963 14391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:49.098012 14376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:49.098130 14376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "edfff45819664869907c7eeeccdd3ac5"
format_stamp: "Formatted at 2026-09-25 02:52:49 on dist-test-slave-252r"
server_key: "601d9e8178a06f2e1e695b4f6cfbf294"
server_key_iv: "3787f0bab7d6f2001c4fa2dfda1ab8c4"
server_key_version: "encryptionkey@0"
I20260925 02:52:49.098222 14376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:49.117491 14376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:49.120090 14376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:49.120337 14376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:49.121671 14376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:49.121735 14376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:49.121793 14376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:49.121833 14376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:52:49 dist-test-slave-252r krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304769, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:49.132058 14376 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:42091
I20260925 02:52:49.132381 14506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:42091 every 8 connection(s)
I20260925 02:52:49.132937 14376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:52:49.134637 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.122528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:53509 (local address 127.12.222.254:33123)
0925 02:52:49.122640 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:49.122644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:49.123663 (+ 1019us) server_negotiation.cc:407] Connection header received
0925 02:52:49.124730 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:49.124735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:49.124803 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:49.124903 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:49.126362 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.126879 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.127914 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.128266 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.128380 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.131241 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:49.131245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:49.131247 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:49.131272 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:49.132813 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:49.133345 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:49.133348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:49.133350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:49.133434 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:49.133785 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:49.133788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:49.133790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:49.133963 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:49.134039 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:49.134421 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 02:52:49.134528 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:52:49.135248 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.123049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:33123 (local address 127.12.222.193:53509)
0925 02:52:49.123531 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:49.123570 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:49.124470 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:49.125103 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:49.125110 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:49.125581 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:49.126180 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.126214 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.127002 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.127006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.127780 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.127788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.128427 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.128431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.128608 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.129125 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:49.129142 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:49.131087 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:49.132927 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:49.132933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:49.132945 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:49.133232 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:49.133541 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:49.133543 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:49.133545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:49.133698 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:49.134047 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:49.134051 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:49.134233 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 02:52:49.134376 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 02:52:49.134576 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":183,"threads_started":1}
I20260925 02:52:49.136437 14507 heartbeater.cc:347] Connected to a master server at 127.12.222.254:33123
I20260925 02:52:49.136579 14507 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:49.137032 14507 heartbeater.cc:511] Master 127.12.222.254:33123 requested a full tablet report, sending...
I20260925 02:52:49.138340 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 14376
I20260925 02:52:49.138468 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:52:49.138561 14301 ts_manager.cc:194] Registered new tserver with Master: edfff45819664869907c7eeeccdd3ac5 (127.12.222.193:42091)
I20260925 02:52:49.138747 13179 external_mini_cluster.cc:1428] Setting key 4a37b4ab528a45043443716546d1d8be
I20260925 02:52:49.139881 14301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:53509
I20260925 02:52:49.146220 14513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.138430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:33123 (local address 127.12.222.193:40725)
0925 02:52:49.139660 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:49.139675 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:49.139769 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:49.140025 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:49.140028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:49.140219 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:49.140371 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.140378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.141207 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.141210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.141764 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.141772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.142216 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.142219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.142308 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.142754 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:49.142769 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:49.143117 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:49.144829 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:49.144832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:49.144837 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:49.145054 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:49.145316 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:49.145319 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:49.145321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:49.145402 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:49.145763 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:49.145767 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:49.145858 (+ 91us) client_negotiation.cc:769] Sending connection context
0925 02:52:49.145969 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 02:52:49.146079 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1155,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260925 02:52:49.146338 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.138476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:40725 (local address 127.12.222.254:33123)
0925 02:52:49.138573 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:49.138579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:49.139749 (+ 1170us) server_negotiation.cc:407] Connection header received
0925 02:52:49.139859 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:49.139862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:49.139907 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:49.139992 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:49.140503 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.141079 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.141886 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.142101 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.142209 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.143259 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:49.143262 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:49.143264 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:49.143297 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:49.144725 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:49.145149 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:49.145152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:49.145154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:49.145227 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:49.145489 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:49.145493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:49.145494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:49.145672 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:49.145758 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:49.146057 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 02:52:49.146190 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
WARNING: no policy specified for kudu/127.12.222.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
I20260925 02:52:49.164965 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:33123
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:33083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:49.266300 14516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:49.266517 14516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:49.266556 14516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:49.270172 14516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:49.270216 14516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:49.270251 14516 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:52:49.270282 14516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:52:49.274983 14516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:33083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.222.254:33123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:49.275825 14516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:49.276700 14516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:49.285599 14524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:49.285584 14522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:49.285609 14521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:49.285796 14516 server_base.cc:1034] running on GCE node
I20260925 02:52:49.286274 14516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:49.286825 14516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:49.287981 14516 hybrid_clock.cc:648] HybridClock initialized: now 1790304769287962 us; error 29 us; skew 500 ppm
Sep 25 02:52:49 dist-test-slave-252r krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304769, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:49.290433 14516 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:52:49.291250 14516 webserver.cc:460] Webserver started at http://127.12.222.194:43795/ using document root <none> and password file <none>
I20260925 02:52:49.291738 14516 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:49.291819 14516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:49.292028 14516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:49.294066 14516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "671f35701ace4ac2a5e0dc4475aeebbf"
format_stamp: "Formatted at 2026-09-25 02:52:49 on dist-test-slave-252r"
server_key: "1998edf4e4e5aed311daa8e50693e09c"
server_key_iv: "4658003f4c85faa1b169a311465498c2"
server_key_version: "encryptionkey@0"
I20260925 02:52:49.294593 14516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "671f35701ace4ac2a5e0dc4475aeebbf"
format_stamp: "Formatted at 2026-09-25 02:52:49 on dist-test-slave-252r"
server_key: "1998edf4e4e5aed311daa8e50693e09c"
server_key_iv: "4658003f4c85faa1b169a311465498c2"
server_key_version: "encryptionkey@0"
I20260925 02:52:49.297917 14516 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:52:49.300019 14531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:49.301180 14516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:49.301306 14516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "671f35701ace4ac2a5e0dc4475aeebbf"
format_stamp: "Formatted at 2026-09-25 02:52:49 on dist-test-slave-252r"
server_key: "1998edf4e4e5aed311daa8e50693e09c"
server_key_iv: "4658003f4c85faa1b169a311465498c2"
server_key_version: "encryptionkey@0"
I20260925 02:52:49.301404 14516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:49.311542 14516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:49.314023 14516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:49.314183 14516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:49.315449 14516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:49.315495 14516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:49.315529 14516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:49.315572 14516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:52:49 dist-test-slave-252r krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304769, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:49.325469 14516 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:38737
I20260925 02:52:49.325798 14648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:38737 every 8 connection(s)
I20260925 02:52:49.326543 14516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:52:49.328317 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.316103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:58315 (local address 127.12.222.254:33123)
0925 02:52:49.316276 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:49.316280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:49.317064 (+ 784us) server_negotiation.cc:407] Connection header received
0925 02:52:49.318104 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:49.318109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:49.318172 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:49.318267 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:49.319506 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.320053 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.321198 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.321418 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.321523 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.324751 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:49.324754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:49.324756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:49.324778 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:49.326339 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:49.326859 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:49.326864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:49.326866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:49.326962 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:49.327439 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:49.327442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:49.327444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:49.327608 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:49.327683 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:49.328074 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 02:52:49.328163 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:52:49.328969 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.316493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:33123 (local address 127.12.222.194:58315)
0925 02:52:49.316908 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:49.316955 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:49.317843 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:49.318473 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:49.318500 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:49.318860 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:49.319317 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.319336 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.320178 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.320181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.321064 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.321073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.321571 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.321575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.321746 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.322319 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:49.322337 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:49.324582 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:49.326441 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:49.326445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:49.326456 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:49.326733 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:49.327067 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:49.327070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:49.327072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:49.327227 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:49.327698 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:49.327702 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:49.327880 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 02:52:49.328016 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 02:52:49.328218 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":125,"threads_started":1}
I20260925 02:52:49.329730 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 14516
I20260925 02:52:49.329839 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:52:49.330039 14649 heartbeater.cc:347] Connected to a master server at 127.12.222.254:33123
I20260925 02:52:49.330106 13179 external_mini_cluster.cc:1428] Setting key 33b2c7dececf84f93bf082cf2cb9cab6
I20260925 02:52:49.330175 14649 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:49.330650 14649 heartbeater.cc:511] Master 127.12.222.254:33123 requested a full tablet report, sending...
I20260925 02:52:49.331532 14301 ts_manager.cc:194] Registered new tserver with Master: 671f35701ace4ac2a5e0dc4475aeebbf (127.12.222.194:38737)
I20260925 02:52:49.332274 14301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:58315
I20260925 02:52:49.338951 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.332313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:41821 (local address 127.12.222.254:33123)
0925 02:52:49.332453 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:49.332457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:49.332472 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:52:49.332505 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:49.332508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:49.332551 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:49.332618 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:49.333213 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.333864 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.334707 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.334946 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.335023 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.336124 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:49.336128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:49.336130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:49.336156 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:49.337476 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:49.337925 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:49.337927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:49.337928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:49.337995 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:49.338266 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:49.338268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:49.338269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:49.338443 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:49.338510 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:49.338786 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 02:52:49.338861 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 02:52:49.339054 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.331942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:33123 (local address 127.12.222.194:41821)
0925 02:52:49.332309 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:49.332325 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:49.332409 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:49.332715 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:49.332717 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:49.332909 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:49.333081 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.333088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.334027 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.334030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.334605 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.334613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.335060 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.335066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.335185 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.335577 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:49.335594 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:49.335975 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:49.337570 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:49.337574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:49.337578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:49.337837 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:49.338084 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:49.338088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:49.338092 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:49.338182 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:49.338534 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:49.338538 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:49.338636 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 02:52:49.338748 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:52:49.338872 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":26,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.12.222.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
I20260925 02:52:49.356521 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:33123
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:33083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:49.456869 14658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:49.457077 14658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:49.457139 14658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:49.460572 14658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:49.460644 14658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:49.460704 14658 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:52:49.460769 14658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:52:49.465214 14658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:33083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.222.254:33123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:49.466004 14658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:49.466823 14658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:49.475145 14666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:49.475149 14664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:49.475157 14663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:49.475332 14658 server_base.cc:1034] running on GCE node
I20260925 02:52:49.475742 14658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:49.476341 14658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:49.477489 14658 hybrid_clock.cc:648] HybridClock initialized: now 1790304769477474 us; error 27 us; skew 500 ppm
Sep 25 02:52:49 dist-test-slave-252r krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304769, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:49.480221 14658 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:52:49.481073 14658 webserver.cc:460] Webserver started at http://127.12.222.195:36469/ using document root <none> and password file <none>
I20260925 02:52:49.481590 14658 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:49.481647 14658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:49.481871 14658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:49.483757 14658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "0b1aebda96e6457392f3312ebba8cc04"
format_stamp: "Formatted at 2026-09-25 02:52:49 on dist-test-slave-252r"
server_key: "01bb6dd40bc8a45bcd05f1a3a3126879"
server_key_iv: "751cf9d40565859afea36f11b25abf82"
server_key_version: "encryptionkey@0"
I20260925 02:52:49.484294 14658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "0b1aebda96e6457392f3312ebba8cc04"
format_stamp: "Formatted at 2026-09-25 02:52:49 on dist-test-slave-252r"
server_key: "01bb6dd40bc8a45bcd05f1a3a3126879"
server_key_iv: "751cf9d40565859afea36f11b25abf82"
server_key_version: "encryptionkey@0"
I20260925 02:52:49.487318 14658 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260925 02:52:49.489393 14673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:49.490383 14658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:52:49.490511 14658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "0b1aebda96e6457392f3312ebba8cc04"
format_stamp: "Formatted at 2026-09-25 02:52:49 on dist-test-slave-252r"
server_key: "01bb6dd40bc8a45bcd05f1a3a3126879"
server_key_iv: "751cf9d40565859afea36f11b25abf82"
server_key_version: "encryptionkey@0"
I20260925 02:52:49.490617 14658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:49.504745 14658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:49.507447 14658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:49.507616 14658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:49.508900 14658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:49.508972 14658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:49.509034 14658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:49.509080 14658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:52:49 dist-test-slave-252r krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304769, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:49.518648 14658 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:45137
I20260925 02:52:49.518713 14789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:45137 every 8 connection(s)
I20260925 02:52:49.519635 14658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:52:49.521070 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.509460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:49209 (local address 127.12.222.254:33123)
0925 02:52:49.509606 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:49.509612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:49.510778 (+ 1166us) server_negotiation.cc:407] Connection header received
0925 02:52:49.511875 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:49.511879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:49.511931 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:49.512005 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:49.513102 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.513620 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.514614 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.514818 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.514917 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.517618 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:49.517624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:49.517627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:49.517655 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:49.519207 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:49.519668 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:49.519674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:49.519677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:49.519767 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:49.520079 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:49.520083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:49.520084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:49.520321 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:49.520423 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:49.520845 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 02:52:49.520959 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:52:49.521464 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 14658
I20260925 02:52:49.521564 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:52:49.521812 13179 external_mini_cluster.cc:1428] Setting key 2b9147fe21e28e71e72fdb8989384253
I20260925 02:52:49.521876 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.510240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:33123 (local address 127.12.222.195:49209)
0925 02:52:49.510635 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:49.510676 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:49.511692 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:49.512166 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:49.512172 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:49.512560 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:49.512967 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.512979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.513737 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.513740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.514498 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.514507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.514975 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.514979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.515137 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.515666 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:49.515680 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:49.517450 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:49.519315 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:49.519319 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:49.519328 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:49.519563 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:49.519853 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:49.519856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:49.519857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:49.519980 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:49.520417 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:49.520422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:49.520642 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 02:52:49.520785 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 02:52:49.521027 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":128,"threads_started":1}
I20260925 02:52:49.523581 14790 heartbeater.cc:347] Connected to a master server at 127.12.222.254:33123
I20260925 02:52:49.523711 14790 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:49.524132 14790 heartbeater.cc:511] Master 127.12.222.254:33123 requested a full tablet report, sending...
I20260925 02:52:49.525065 14300 ts_manager.cc:194] Registered new tserver with Master: 0b1aebda96e6457392f3312ebba8cc04 (127.12.222.195:45137)
I20260925 02:52:49.525542 14300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:49209
I20260925 02:52:49.531505 14796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.524793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:33123 (local address 127.12.222.195:40199)
0925 02:52:49.525842 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:49.525853 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:49.525972 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:49.526182 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:49.526185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:49.526382 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:49.526484 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.526491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.527156 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.527159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.527723 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.527731 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.528106 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.528109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.528170 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.528641 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:49.528658 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:49.528944 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:49.530283 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:49.530288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:49.530293 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:49.530477 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:49.530696 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:49.530698 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:49.530718 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:49.530787 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:49.531091 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:49.531093 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:49.531189 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 02:52:49.531284 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 02:52:49.531375 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"thread_start_us":92,"threads_started":1}
I20260925 02:52:49.531514 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.524874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:40199 (local address 127.12.222.254:33123)
0925 02:52:49.524973 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:49.524976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:49.525928 (+ 952us) server_negotiation.cc:407] Connection header received
0925 02:52:49.526050 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:49.526052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:49.526087 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:49.526148 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:49.526594 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.527037 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.527827 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.528021 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.528090 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.529057 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:49.529059 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:49.529060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:49.529078 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:49.530181 (+ 1103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:49.530555 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:49.530557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:49.530558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:49.530624 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:49.530861 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:49.530863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:49.530864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:49.531015 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:49.531090 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:49.531336 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 02:52:49.531414 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:52:49.535701 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:52:49 dist-test-slave-252r krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304769, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:52:49 dist-test-slave-252r krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304769, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:49.558605 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.551743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53698 (local address 127.12.222.254:33123)
0925 02:52:49.551851 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:49.551854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:49.551865 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:52:49.551892 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:49.551895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:49.551925 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:49.551994 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:49.552659 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.553129 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.553834 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.554015 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.554078 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.556085 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:49.556087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:49.556089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:49.556108 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:49.557459 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:49.557783 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:49.557785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:49.557786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:49.557844 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:49.558028 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:49.558030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:49.558030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:49.558152 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:49.558252 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:49.558399 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 02:52:49.558467 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:52:49.560810 14300 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:52:49.563514 14300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:52:49.577564 14810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.572624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.12.222.193:42091)
0925 02:52:49.572940 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:49.572959 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:49.572982 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:52:49.573060 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:49.573065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:49.573272 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:49.573459 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:49.573725 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.574608 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.576180 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.576938 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.577087 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.577212 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:49.577337 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 02:52:49.577419 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":98,"threads_started":1}
I20260925 02:52:49.577574 14809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.571812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:42091 (local address 127.0.0.1:52908)
0925 02:52:49.572543 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:49.572554 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:49.572623 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:49.573457 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:49.573463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:49.573487 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:49.573616 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.573624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.574904 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.574906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.576037 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.576046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.577059 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.577063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.577168 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.577206 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:52:49.577315 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 02:52:49.577425 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260925 02:52:49.578887 14807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.571162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:38737 (local address 127.0.0.1:43020)
0925 02:52:49.571715 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:49.571762 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:49.571883 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:49.574105 (+ 2222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:49.574108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:49.574125 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:49.574216 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.574224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.575606 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.575610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.576797 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.576806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.578507 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.578511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.578614 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.578637 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:52:49.578699 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 02:52:49.578760 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":28,"thread_start_us":66,"threads_started":1}
I20260925 02:52:49.579505 14811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.573064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.12.222.195:45137)
0925 02:52:49.573537 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:49.573549 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:49.574553 (+ 1004us) server_negotiation.cc:407] Connection header received
0925 02:52:49.574658 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:49.574663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:49.574836 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:49.574982 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:49.575342 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.576493 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.578096 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.578867 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.579043 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.579151 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:49.579246 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 02:52:49.579332 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":90,"threads_started":1}
I20260925 02:52:49.579423 14808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.572313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:45137 (local address 127.0.0.1:54920)
0925 02:52:49.574343 (+ 2030us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:49.574358 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:49.574438 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:49.575122 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:49.575125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:49.575141 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:49.575229 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.575233 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.576623 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.576626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.577879 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.577889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.579016 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.579020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.579110 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.579132 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:52:49.579219 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 02:52:49.579280 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1966,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260925 02:52:49.579531 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.573253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43020 (local address 127.12.222.194:38737)
0925 02:52:49.573753 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:49.573763 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:49.573790 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:52:49.573850 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:49.573853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:49.573988 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:49.574126 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:49.574371 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.575479 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.576909 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.578264 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.578396 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.579237 (+ 841us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:49.579282 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 02:52:49.579380 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":124,"threads_started":1}
I20260925 02:52:49.580561 14441 tablet_service.cc:1467] Processing CreateTablet for tablet 0880e56a33e648af9c462aaf79f6fe58 (DEFAULT_TABLE table=test-table [id=867d2b102dc84d5a9c07894dd2e8d9bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:49.581660 14441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0880e56a33e648af9c462aaf79f6fe58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:49.582159 14723 tablet_service.cc:1467] Processing CreateTablet for tablet 0880e56a33e648af9c462aaf79f6fe58 (DEFAULT_TABLE table=test-table [id=867d2b102dc84d5a9c07894dd2e8d9bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:49.582405 14582 tablet_service.cc:1467] Processing CreateTablet for tablet 0880e56a33e648af9c462aaf79f6fe58 (DEFAULT_TABLE table=test-table [id=867d2b102dc84d5a9c07894dd2e8d9bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:49.583012 14723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0880e56a33e648af9c462aaf79f6fe58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:49.583374 14582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0880e56a33e648af9c462aaf79f6fe58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:49.589449 14813 tablet_bootstrap.cc:492] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5: Bootstrap starting.
I20260925 02:52:49.590636 14814 tablet_bootstrap.cc:492] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04: Bootstrap starting.
I20260925 02:52:49.591555 14813 tablet_bootstrap.cc:654] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:49.592552 14813 log.cc:824] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:49.592641 14815 tablet_bootstrap.cc:492] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf: Bootstrap starting.
I20260925 02:52:49.592789 14814 tablet_bootstrap.cc:654] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:49.593637 14814 log.cc:824] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:49.594511 14813 tablet_bootstrap.cc:492] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5: No bootstrap required, opened a new log
I20260925 02:52:49.594693 14815 tablet_bootstrap.cc:654] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:49.594692 14813 ts_tablet_manager.cc:1397] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260925 02:52:49.595460 14814 tablet_bootstrap.cc:492] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04: No bootstrap required, opened a new log
I20260925 02:52:49.595458 14815 log.cc:824] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:49.595634 14814 ts_tablet_manager.cc:1397] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:52:49.597247 14815 tablet_bootstrap.cc:492] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf: No bootstrap required, opened a new log
I20260925 02:52:49.597421 14815 ts_tablet_manager.cc:1397] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:52:49.598011 14813 raft_consensus.cc:348] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:49.598219 14813 raft_consensus.cc:374] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:49.598270 14813 raft_consensus.cc:729] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edfff45819664869907c7eeeccdd3ac5, State: Initialized, Role: FOLLOWER
I20260925 02:52:49.598672 14813 consensus_queue.cc:260] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:49.599202 14507 heartbeater.cc:503] Master 127.12.222.254:33123 was elected leader, sending a full tablet report...
I20260925 02:52:49.599519 14813 ts_tablet_manager.cc:1428] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260925 02:52:49.599792 14814 raft_consensus.cc:348] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:49.599989 14814 raft_consensus.cc:374] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:49.600044 14814 raft_consensus.cc:729] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b1aebda96e6457392f3312ebba8cc04, State: Initialized, Role: FOLLOWER
I20260925 02:52:49.600448 14814 consensus_queue.cc:260] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:49.601044 14790 heartbeater.cc:503] Master 127.12.222.254:33123 was elected leader, sending a full tablet report...
I20260925 02:52:49.601131 14815 raft_consensus.cc:348] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:49.601347 14815 raft_consensus.cc:374] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:49.601368 14814 ts_tablet_manager.cc:1428] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 02:52:49.601393 14815 raft_consensus.cc:729] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671f35701ace4ac2a5e0dc4475aeebbf, State: Initialized, Role: FOLLOWER
I20260925 02:52:49.601749 14815 consensus_queue.cc:260] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:49.602953 14649 heartbeater.cc:503] Master 127.12.222.254:33123 was elected leader, sending a full tablet report...
I20260925 02:52:49.603093 14815 ts_tablet_manager.cc:1428] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260925 02:52:49.634064 14508 tablet.cc:2369] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:52:49.653110 14821 raft_consensus.cc:482] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:52:49.653262 14821 raft_consensus.cc:504] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:49.654234 14821 leader_election.cc:288] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edfff45819664869907c7eeeccdd3ac5 (127.12.222.193:42091), 0b1aebda96e6457392f3312ebba8cc04 (127.12.222.195:45137)
I20260925 02:52:49.658075 14811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.654543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:39243 (local address 127.12.222.195:45137)
0925 02:52:49.654668 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:49.654671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:49.654703 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:52:49.654746 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:49.654748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:49.654784 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:49.654838 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:49.655147 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.655832 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.656985 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.657545 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.657672 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.657737 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:49.657914 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 02:52:49.657983 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:52:49.658082 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.654432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:45137 (local address 127.12.222.194:39243)
0925 02:52:49.654543 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:49.654559 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:49.654643 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:49.654876 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:49.654879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:49.654898 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:49.655022 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.655027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.655939 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.655942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.656849 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.656858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.657648 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.657668 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.657760 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.657802 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 02:52:49.657893 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 02:52:49.657963 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 02:52:49.658670 14743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0880e56a33e648af9c462aaf79f6fe58" candidate_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b1aebda96e6457392f3312ebba8cc04" is_pre_election: true
I20260925 02:52:49.658941 14822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.654426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:42091 (local address 127.12.222.194:54993)
0925 02:52:49.655061 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:49.655092 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:49.655192 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:49.655395 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:49.655398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:49.655415 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:49.655523 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.655528 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.656325 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.656328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.657712 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:49.657721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.658514 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.658518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:49.658634 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.658658 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:52:49.658747 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 02:52:49.658812 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":260,"thread_start_us":125,"threads_started":1}
I20260925 02:52:49.658926 14810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.654465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:54993 (local address 127.12.222.193:42091)
0925 02:52:49.654572 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:49.654578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:49.655170 (+ 592us) server_negotiation.cc:407] Connection header received
0925 02:52:49.655270 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:49.655273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:49.655309 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:49.655361 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:49.655619 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.656205 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.657823 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.658388 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.658514 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.658557 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:49.658764 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 02:52:49.658826 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:52:49.658934 14743 raft_consensus.cc:2454] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 671f35701ace4ac2a5e0dc4475aeebbf in term 0.
I20260925 02:52:49.659443 14532 leader_election.cc:302] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b1aebda96e6457392f3312ebba8cc04, 671f35701ace4ac2a5e0dc4475aeebbf; no voters:
I20260925 02:52:49.659438 14461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0880e56a33e648af9c462aaf79f6fe58" candidate_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edfff45819664869907c7eeeccdd3ac5" is_pre_election: true
I20260925 02:52:49.659693 14461 raft_consensus.cc:2454] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 671f35701ace4ac2a5e0dc4475aeebbf in term 0.
I20260925 02:52:49.659689 14821 raft_consensus.cc:2781] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:52:49.659773 14821 raft_consensus.cc:482] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:52:49.659814 14821 raft_consensus.cc:3037] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:49.660869 14821 raft_consensus.cc:504] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:49.661151 14821 leader_election.cc:288] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [CANDIDATE]: Term 1 election: Requested vote from peers edfff45819664869907c7eeeccdd3ac5 (127.12.222.193:42091), 0b1aebda96e6457392f3312ebba8cc04 (127.12.222.195:45137)
I20260925 02:52:49.661446 14461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0880e56a33e648af9c462aaf79f6fe58" candidate_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edfff45819664869907c7eeeccdd3ac5"
I20260925 02:52:49.661556 14461 raft_consensus.cc:3037] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:49.662503 14461 raft_consensus.cc:2454] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 671f35701ace4ac2a5e0dc4475aeebbf in term 1.
I20260925 02:52:49.662746 14743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0880e56a33e648af9c462aaf79f6fe58" candidate_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b1aebda96e6457392f3312ebba8cc04"
I20260925 02:52:49.662811 14533 leader_election.cc:302] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 671f35701ace4ac2a5e0dc4475aeebbf, edfff45819664869907c7eeeccdd3ac5; no voters:
I20260925 02:52:49.662866 14743 raft_consensus.cc:3037] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:49.663003 14821 raft_consensus.cc:2781] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:49.663214 14821 raft_consensus.cc:686] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [term 1 LEADER]: Becoming Leader. State: Replica: 671f35701ace4ac2a5e0dc4475aeebbf, State: Running, Role: LEADER
I20260925 02:52:49.663548 14821 consensus_queue.cc:237] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:49.664140 14743 raft_consensus.cc:2454] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 671f35701ace4ac2a5e0dc4475aeebbf in term 1.
I20260925 02:52:49.666610 14300 catalog_manager.cc:5495] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 671f35701ace4ac2a5e0dc4475aeebbf (127.12.222.194). New cstate: current_term: 1 leader_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } health_report { overall_health: UNKNOWN } } }
I20260925 02:52:49.704216 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:49.701012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.12.222.194:38737)
0925 02:52:49.701147 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:49.701152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:49.701166 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:52:49.701352 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:49.701354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:49.701403 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:49.701479 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:49.701695 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.702185 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.703003 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:49.703193 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:49.703263 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:49.703529 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:49.703875 (+ 346us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:49.703956 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:49.704015 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 02:52:49.704058 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:52:49.709815 14743 raft_consensus.cc:1261] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04 [term 1 FOLLOWER]: Refusing update from remote peer 671f35701ace4ac2a5e0dc4475aeebbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:49.710546 14823 consensus_queue.cc:1035] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:49.710790 14461 raft_consensus.cc:1261] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5 [term 1 FOLLOWER]: Refusing update from remote peer 671f35701ace4ac2a5e0dc4475aeebbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:49.711336 14823 consensus_queue.cc:1035] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:49.718266 14828 mvcc.cc:205] Tried to move back new op lower bound from 7333088336723517440 to 7333088336544821248. Current Snapshot: MvccSnapshot[applied={T|T < 7333088336723517440}]
I20260925 02:52:49.719019 14827 mvcc.cc:205] Tried to move back new op lower bound from 7333088336723517440 to 7333088336544821248. Current Snapshot: MvccSnapshot[applied={T|T < 7333088336723517440}]
I20260925 02:52:49.729146 14302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:52:49.729390 14302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:52:49.732218 14302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447: Sending DeleteTablet for 3 replicas of tablet 0880e56a33e648af9c462aaf79f6fe58
I20260925 02:52:49.733078 14441 tablet_service.cc:1514] Processing DeleteTablet for tablet 0880e56a33e648af9c462aaf79f6fe58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:49 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:52908
I20260925 02:52:49.733647 14836 tablet_replica.cc:330] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5: stopping tablet replica
I20260925 02:52:49.734000 14836 raft_consensus.cc:2229] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:52:49.734071 14302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 02:52:49.734470 14836 raft_consensus.cc:2258] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 02:52:49.734529 14302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:52:49.735350 14582 tablet_service.cc:1514] Processing DeleteTablet for tablet 0880e56a33e648af9c462aaf79f6fe58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:49 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:43020
I20260925 02:52:49.735872 14723 tablet_service.cc:1514] Processing DeleteTablet for tablet 0880e56a33e648af9c462aaf79f6fe58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:49 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:54920
I20260925 02:52:49.735966 14836 ts_tablet_manager.cc:1905] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:49.736325 14837 tablet_replica.cc:330] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf: stopping tablet replica
I20260925 02:52:49.736632 14837 raft_consensus.cc:2229] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:52:49.737020 14837 raft_consensus.cc:2258] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:49.738418 14838 tablet_replica.cc:330] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04: stopping tablet replica
I20260925 02:52:49.738684 14838 raft_consensus.cc:2229] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:52:49.738962 14838 raft_consensus.cc:2258] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:49.739162 14836 ts_tablet_manager.cc:1918] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:52:49.739289 14836 log.cc:1196] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/wal/wals/0880e56a33e648af9c462aaf79f6fe58
I20260925 02:52:49.739604 14723 tablet_service.cc:1467] Processing CreateTablet for tablet 4af75471fb7b4312a1b3313d76a030c7 (DEFAULT_TABLE table=test-table [id=3a832ec0ba04472b8494b9f7a68f981a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:49.739686 14582 tablet_service.cc:1467] Processing CreateTablet for tablet 4af75471fb7b4312a1b3313d76a030c7 (DEFAULT_TABLE table=test-table [id=3a832ec0ba04472b8494b9f7a68f981a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:49.739663 14836 ts_tablet_manager.cc:1939] T 0880e56a33e648af9c462aaf79f6fe58 P edfff45819664869907c7eeeccdd3ac5: Deleting consensus metadata
I20260925 02:52:49.739847 14723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4af75471fb7b4312a1b3313d76a030c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:49.739883 14582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4af75471fb7b4312a1b3313d76a030c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:49.742210 14815 tablet_bootstrap.cc:492] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf: Bootstrap starting.
I20260925 02:52:49.742324 14287 catalog_manager.cc:4841] TS edfff45819664869907c7eeeccdd3ac5 (127.12.222.193:42091): tablet 0880e56a33e648af9c462aaf79f6fe58 (table test-table [id=867d2b102dc84d5a9c07894dd2e8d9bb]) successfully deleted
I20260925 02:52:49.742833 14441 tablet_service.cc:1467] Processing CreateTablet for tablet 4af75471fb7b4312a1b3313d76a030c7 (DEFAULT_TABLE table=test-table [id=3a832ec0ba04472b8494b9f7a68f981a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:49.743059 14441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4af75471fb7b4312a1b3313d76a030c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:49.743078 14815 tablet_bootstrap.cc:654] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:49.743733 14814 tablet_bootstrap.cc:492] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04: Bootstrap starting.
I20260925 02:52:49.743918 14837 ts_tablet_manager.cc:1905] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:49.744742 14814 tablet_bootstrap.cc:654] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:49.745204 14813 tablet_bootstrap.cc:492] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5: Bootstrap starting.
I20260925 02:52:49.745823 14838 ts_tablet_manager.cc:1905] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:49.746102 14813 tablet_bootstrap.cc:654] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:49.747608 14837 ts_tablet_manager.cc:1918] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:52:49.747628 14813 tablet_bootstrap.cc:492] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5: No bootstrap required, opened a new log
I20260925 02:52:49.747694 14837 log.cc:1196] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/0880e56a33e648af9c462aaf79f6fe58
I20260925 02:52:49.747711 14813 ts_tablet_manager.cc:1397] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:52:49.747977 14837 ts_tablet_manager.cc:1939] T 0880e56a33e648af9c462aaf79f6fe58 P 671f35701ace4ac2a5e0dc4475aeebbf: Deleting consensus metadata
I20260925 02:52:49.748098 14838 ts_tablet_manager.cc:1918] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:52:49.748178 14838 log.cc:1196] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/0880e56a33e648af9c462aaf79f6fe58
I20260925 02:52:49.748138 14813 raft_consensus.cc:348] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:49.748271 14813 raft_consensus.cc:374] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:49.748307 14813 raft_consensus.cc:729] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edfff45819664869907c7eeeccdd3ac5, State: Initialized, Role: FOLLOWER
I20260925 02:52:49.748495 14838 ts_tablet_manager.cc:1939] T 0880e56a33e648af9c462aaf79f6fe58 P 0b1aebda96e6457392f3312ebba8cc04: Deleting consensus metadata
I20260925 02:52:49.748451 14813 consensus_queue.cc:260] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:49.748708 14813 ts_tablet_manager.cc:1428] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:52:49.748844 14287 catalog_manager.cc:4841] TS 671f35701ace4ac2a5e0dc4475aeebbf (127.12.222.194:38737): tablet 0880e56a33e648af9c462aaf79f6fe58 (table test-table [id=867d2b102dc84d5a9c07894dd2e8d9bb]) successfully deleted
I20260925 02:52:49.749172 14815 tablet_bootstrap.cc:492] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf: No bootstrap required, opened a new log
I20260925 02:52:49.749243 14815 ts_tablet_manager.cc:1397] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 02:52:49.749542 14286 catalog_manager.cc:4841] TS 0b1aebda96e6457392f3312ebba8cc04 (127.12.222.195:45137): tablet 0880e56a33e648af9c462aaf79f6fe58 (table test-table [id=867d2b102dc84d5a9c07894dd2e8d9bb]) successfully deleted
I20260925 02:52:49.749702 14815 raft_consensus.cc:348] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:49.749799 14815 raft_consensus.cc:374] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:49.749833 14815 raft_consensus.cc:729] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671f35701ace4ac2a5e0dc4475aeebbf, State: Initialized, Role: FOLLOWER
I20260925 02:52:49.749938 14814 tablet_bootstrap.cc:492] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04: No bootstrap required, opened a new log
I20260925 02:52:49.750005 14814 ts_tablet_manager.cc:1397] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 02:52:49.749958 14815 consensus_queue.cc:260] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:49.750211 14815 ts_tablet_manager.cc:1428] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:52:49.750471 14814 raft_consensus.cc:348] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:49.750584 14814 raft_consensus.cc:374] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:49.750625 14814 raft_consensus.cc:729] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b1aebda96e6457392f3312ebba8cc04, State: Initialized, Role: FOLLOWER
I20260925 02:52:49.750737 14814 consensus_queue.cc:260] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:49.751004 14814 ts_tablet_manager.cc:1428] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260925 02:52:49.770613 14791 tablet.cc:2369] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:52:49.827572 14650 tablet.cc:2369] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:52:50.083686 14820 raft_consensus.cc:482] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:52:50.083859 14820 raft_consensus.cc:504] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:50.085067 14820 leader_election.cc:288] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edfff45819664869907c7eeeccdd3ac5 (127.12.222.193:42091), 671f35701ace4ac2a5e0dc4475aeebbf (127.12.222.194:38737)
I20260925 02:52:50.089174 14810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:50.085367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:44885 (local address 127.12.222.193:42091)
0925 02:52:50.085553 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:50.085559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:50.085675 (+ 116us) server_negotiation.cc:407] Connection header received
0925 02:52:50.085916 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:50.085920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:50.086137 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:50.086225 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:50.086537 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.087096 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.088138 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.088589 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.088709 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:50.088761 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:50.089002 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 02:52:50.089067 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 02:52:50.089167 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:50.085745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:59093 (local address 127.12.222.194:38737)
0925 02:52:50.085887 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:50.085891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:50.086144 (+ 253us) server_negotiation.cc:407] Connection header received
0925 02:52:50.086285 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:50.086288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:50.086335 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:50.086417 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:50.086669 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.087170 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.088167 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.088589 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.088707 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:50.088762 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:50.088969 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 02:52:50.089041 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:52:50.089190 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:50.085628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:38737 (local address 127.12.222.195:59093)
0925 02:52:50.085944 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:50.085970 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:50.086184 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:50.086433 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:50.086435 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:50.086448 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:50.086567 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:50.086573 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.087283 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.087285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:50.088025 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:50.088033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.088700 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.088703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:50.088845 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:50.088867 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:52:50.088926 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 02:52:50.089012 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":58,"thread_start_us":55,"threads_started":1}
I20260925 02:52:50.089191 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:50.085256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:42091 (local address 127.12.222.195:44885)
0925 02:52:50.085530 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:50.085543 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:50.085768 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:50.086260 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:50.086263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:50.086283 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:50.086418 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:50.086425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.087195 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.087202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:50.088017 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:50.088026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.088703 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.088706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:50.088871 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:50.088887 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:52:50.088975 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 02:52:50.089041 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":35,"thread_start_us":66,"threads_started":1}
I20260925 02:52:50.089700 14602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af75471fb7b4312a1b3313d76a030c7" candidate_uuid: "0b1aebda96e6457392f3312ebba8cc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" is_pre_election: true
I20260925 02:52:50.089808 14461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af75471fb7b4312a1b3313d76a030c7" candidate_uuid: "0b1aebda96e6457392f3312ebba8cc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edfff45819664869907c7eeeccdd3ac5" is_pre_election: true
I20260925 02:52:50.089942 14602 raft_consensus.cc:2454] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b1aebda96e6457392f3312ebba8cc04 in term 0.
I20260925 02:52:50.089962 14461 raft_consensus.cc:2454] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b1aebda96e6457392f3312ebba8cc04 in term 0.
I20260925 02:52:50.090305 14675 leader_election.cc:302] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b1aebda96e6457392f3312ebba8cc04, edfff45819664869907c7eeeccdd3ac5; no voters:
I20260925 02:52:50.090525 14820 raft_consensus.cc:2781] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:52:50.090585 14820 raft_consensus.cc:482] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:52:50.090622 14820 raft_consensus.cc:3037] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:50.091544 14820 raft_consensus.cc:504] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:50.091827 14820 leader_election.cc:288] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [CANDIDATE]: Term 1 election: Requested vote from peers edfff45819664869907c7eeeccdd3ac5 (127.12.222.193:42091), 671f35701ace4ac2a5e0dc4475aeebbf (127.12.222.194:38737)
I20260925 02:52:50.092114 14461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af75471fb7b4312a1b3313d76a030c7" candidate_uuid: "0b1aebda96e6457392f3312ebba8cc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edfff45819664869907c7eeeccdd3ac5"
I20260925 02:52:50.092176 14602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af75471fb7b4312a1b3313d76a030c7" candidate_uuid: "0b1aebda96e6457392f3312ebba8cc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671f35701ace4ac2a5e0dc4475aeebbf"
I20260925 02:52:50.092270 14461 raft_consensus.cc:3037] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:50.092327 14602 raft_consensus.cc:3037] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:50.093263 14461 raft_consensus.cc:2454] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b1aebda96e6457392f3312ebba8cc04 in term 1.
I20260925 02:52:50.093278 14602 raft_consensus.cc:2454] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b1aebda96e6457392f3312ebba8cc04 in term 1.
I20260925 02:52:50.093541 14675 leader_election.cc:302] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b1aebda96e6457392f3312ebba8cc04, 671f35701ace4ac2a5e0dc4475aeebbf; no voters:
I20260925 02:52:50.093734 14820 raft_consensus.cc:2781] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:50.093899 14820 raft_consensus.cc:686] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [term 1 LEADER]: Becoming Leader. State: Replica: 0b1aebda96e6457392f3312ebba8cc04, State: Running, Role: LEADER
I20260925 02:52:50.094130 14820 consensus_queue.cc:237] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:50.096361 14301 catalog_manager.cc:5495] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b1aebda96e6457392f3312ebba8cc04 (127.12.222.195). New cstate: current_term: 1 leader_uuid: "0b1aebda96e6457392f3312ebba8cc04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } health_report { overall_health: HEALTHY } } }
Sep 25 02:52:50 dist-test-slave-252r krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304768, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:50.133292 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:50.125518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:33123 (local address 127.0.0.1:53708)
0925 02:52:50.125771 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:50.125782 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:50.125856 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:50.126091 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:50.126093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:50.126502 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:50.126583 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:50.126589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.127205 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.127208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:50.127736 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:50.127742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.128108 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.128110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:50.128167 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:50.128813 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:50.128824 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:50.130104 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:50.131654 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:50.131662 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:50.131666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:50.131988 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:50.132260 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:50.132263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:50.132265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:50.132357 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:50.132701 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:50.132708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:50.132825 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 02:52:50.132945 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 02:52:50.133098 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":76,"threads_started":1}
I20260925 02:52:50.133306 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:50.125563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.12.222.254:33123)
0925 02:52:50.125811 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:50.125817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:50.125829 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:52:50.125932 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:50.125935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:50.125987 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:50.126056 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:50.126674 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.127126 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.127823 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.128025 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.128091 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:50.130223 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:50.130226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:50.130228 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:50.130246 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:50.131560 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:50.132080 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:50.132083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:50.132084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:50.132159 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:50.132440 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:50.132443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:50.132444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:50.132599 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:50.132674 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:50.132994 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 02:52:50.133167 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":198,"thread_start_us":113,"threads_started":1}
I20260925 02:52:50.137086 14300 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:53708:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:52:50.137804 14300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:52:50.142107 14441 tablet_service.cc:1467] Processing CreateTablet for tablet a9595d4f7b7a4a2b8f2a7313456b5b7b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1cc04d0bf35f4143be773c3faae67792]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:52:50.142275 14723 tablet_service.cc:1467] Processing CreateTablet for tablet a9595d4f7b7a4a2b8f2a7313456b5b7b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1cc04d0bf35f4143be773c3faae67792]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:52:50.142469 14441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9595d4f7b7a4a2b8f2a7313456b5b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:50.142597 14723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9595d4f7b7a4a2b8f2a7313456b5b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:50.145392 14814 tablet_bootstrap.cc:492] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04: Bootstrap starting.
I20260925 02:52:50.145668 14582 tablet_service.cc:1467] Processing CreateTablet for tablet a9595d4f7b7a4a2b8f2a7313456b5b7b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1cc04d0bf35f4143be773c3faae67792]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:52:50.146037 14582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9595d4f7b7a4a2b8f2a7313456b5b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:50.146433 14814 tablet_bootstrap.cc:654] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:50.147037 14813 tablet_bootstrap.cc:492] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5: Bootstrap starting.
I20260925 02:52:50.147883 14813 tablet_bootstrap.cc:654] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:50.148515 14815 tablet_bootstrap.cc:492] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf: Bootstrap starting.
I20260925 02:52:50.148761 14814 tablet_bootstrap.cc:492] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04: No bootstrap required, opened a new log
I20260925 02:52:50.148815 14814 ts_tablet_manager.cc:1397] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:52:50.149159 14814 raft_consensus.cc:348] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:50.149230 14814 raft_consensus.cc:374] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:50.149250 14814 raft_consensus.cc:729] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b1aebda96e6457392f3312ebba8cc04, State: Initialized, Role: FOLLOWER
I20260925 02:52:50.149364 14814 consensus_queue.cc:260] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:50.149413 14815 tablet_bootstrap.cc:654] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:50.149641 14814 ts_tablet_manager.cc:1428] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:52:50.149585 14820 tablet_replica.cc:439] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } } }
I20260925 02:52:50.149763 14820 tablet_replica.cc:442] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:52:50.150301 14813 tablet_bootstrap.cc:492] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5: No bootstrap required, opened a new log
I20260925 02:52:50.150388 14813 ts_tablet_manager.cc:1397] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:52:50.150874 14813 raft_consensus.cc:348] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:50.150996 14813 raft_consensus.cc:374] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:50.151032 14813 raft_consensus.cc:729] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edfff45819664869907c7eeeccdd3ac5, State: Initialized, Role: FOLLOWER
I20260925 02:52:50.151152 14813 consensus_queue.cc:260] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:50.151444 14813 ts_tablet_manager.cc:1428] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:52:50.151443 14819 tablet_replica.cc:439] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } } }
I20260925 02:52:50.152145 14819 tablet_replica.cc:442] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:52:50.152530 14815 tablet_bootstrap.cc:492] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf: No bootstrap required, opened a new log
I20260925 02:52:50.152613 14815 ts_tablet_manager.cc:1397] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 02:52:50.152954 14815 raft_consensus.cc:348] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:50.153059 14815 raft_consensus.cc:374] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:50.153088 14815 raft_consensus.cc:729] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671f35701ace4ac2a5e0dc4475aeebbf, State: Initialized, Role: FOLLOWER
I20260925 02:52:50.153196 14815 consensus_queue.cc:260] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:50.153422 14831 tablet_replica.cc:439] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } } }
I20260925 02:52:50.153568 14831 tablet_replica.cc:442] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:52:50.153432 14815 ts_tablet_manager.cc:1428] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:52:50.198781 14820 raft_consensus.cc:482] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:52:50.198901 14820 raft_consensus.cc:504] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:50.199311 14820 leader_election.cc:288] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edfff45819664869907c7eeeccdd3ac5 (127.12.222.193:42091), 671f35701ace4ac2a5e0dc4475aeebbf (127.12.222.194:38737)
I20260925 02:52:50.199662 14461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9595d4f7b7a4a2b8f2a7313456b5b7b" candidate_uuid: "0b1aebda96e6457392f3312ebba8cc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edfff45819664869907c7eeeccdd3ac5" is_pre_election: true
I20260925 02:52:50.199718 14602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9595d4f7b7a4a2b8f2a7313456b5b7b" candidate_uuid: "0b1aebda96e6457392f3312ebba8cc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" is_pre_election: true
I20260925 02:52:50.199808 14461 raft_consensus.cc:2454] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b1aebda96e6457392f3312ebba8cc04 in term 0.
I20260925 02:52:50.199862 14602 raft_consensus.cc:2454] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b1aebda96e6457392f3312ebba8cc04 in term 0.
I20260925 02:52:50.200114 14675 leader_election.cc:302] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b1aebda96e6457392f3312ebba8cc04, edfff45819664869907c7eeeccdd3ac5; no voters:
I20260925 02:52:50.200300 14820 raft_consensus.cc:2781] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:52:50.200361 14820 raft_consensus.cc:482] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:52:50.200398 14820 raft_consensus.cc:3037] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:50.201362 14820 raft_consensus.cc:504] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:50.201650 14820 leader_election.cc:288] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [CANDIDATE]: Term 1 election: Requested vote from peers edfff45819664869907c7eeeccdd3ac5 (127.12.222.193:42091), 671f35701ace4ac2a5e0dc4475aeebbf (127.12.222.194:38737)
I20260925 02:52:50.202207 14461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9595d4f7b7a4a2b8f2a7313456b5b7b" candidate_uuid: "0b1aebda96e6457392f3312ebba8cc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edfff45819664869907c7eeeccdd3ac5"
I20260925 02:52:50.202345 14461 raft_consensus.cc:3037] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:50.202315 14602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9595d4f7b7a4a2b8f2a7313456b5b7b" candidate_uuid: "0b1aebda96e6457392f3312ebba8cc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671f35701ace4ac2a5e0dc4475aeebbf"
I20260925 02:52:50.202442 14602 raft_consensus.cc:3037] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:50.203470 14602 raft_consensus.cc:2454] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b1aebda96e6457392f3312ebba8cc04 in term 1.
I20260925 02:52:50.203470 14461 raft_consensus.cc:2454] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b1aebda96e6457392f3312ebba8cc04 in term 1.
I20260925 02:52:50.203744 14675 leader_election.cc:302] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b1aebda96e6457392f3312ebba8cc04, 671f35701ace4ac2a5e0dc4475aeebbf; no voters:
I20260925 02:52:50.203915 14820 raft_consensus.cc:2781] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:50.203995 14820 raft_consensus.cc:686] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [term 1 LEADER]: Becoming Leader. State: Replica: 0b1aebda96e6457392f3312ebba8cc04, State: Running, Role: LEADER
I20260925 02:52:50.204180 14820 consensus_queue.cc:237] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } }
I20260925 02:52:50.204802 14844 tablet_replica.cc:439] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04: TxnStatusTablet state changed. Reason: New leader 0b1aebda96e6457392f3312ebba8cc04. Latest consensus state: current_term: 1 leader_uuid: "0b1aebda96e6457392f3312ebba8cc04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } } }
I20260925 02:52:50.204929 14844 tablet_replica.cc:442] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:52:50.205206 14857 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:52:50.205389 14857 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:52:50.205739 14301 catalog_manager.cc:5495] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b1aebda96e6457392f3312ebba8cc04 (127.12.222.195). New cstate: current_term: 1 leader_uuid: "0b1aebda96e6457392f3312ebba8cc04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } health_report { overall_health: HEALTHY } } }
I20260925 02:52:50.227350 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:50.223783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:45137 (local address 127.0.0.1:54934)
0925 02:52:50.223907 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:50.223921 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:50.224030 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:50.224348 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:50.224351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:50.224358 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:52:50.224454 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:50.224460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.225259 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.225263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:50.225844 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:50.225851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.226305 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.226309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:50.226399 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:50.226437 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:52:50.227027 (+ 590us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:52:50.227043 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:52:50.227127 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 02:52:50.227236 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 02:52:50.227367 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:50.223859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54934 (local address 127.12.222.195:45137)
0925 02:52:50.224058 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:50.224063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:50.224078 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:52:50.224142 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:50.224145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:50.224202 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:50.224336 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:50.224559 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.225142 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.225953 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.226196 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.226294 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:50.226571 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:50.226924 (+ 353us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:50.227041 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:50.227139 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 02:52:50.227201 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":58,"threads_started":1}
I20260925 02:52:50.230137 14461 raft_consensus.cc:1261] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5 [term 1 FOLLOWER]: Refusing update from remote peer 0b1aebda96e6457392f3312ebba8cc04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:50.230221 14602 raft_consensus.cc:1261] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf [term 1 FOLLOWER]: Refusing update from remote peer 0b1aebda96e6457392f3312ebba8cc04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:50.230579 14844 consensus_queue.cc:1035] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:50.230716 14820 consensus_queue.cc:1035] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:50.232069 14819 tablet_replica.cc:439] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5: TxnStatusTablet state changed. Reason: New leader 0b1aebda96e6457392f3312ebba8cc04. Latest consensus state: current_term: 1 leader_uuid: "0b1aebda96e6457392f3312ebba8cc04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } } }
I20260925 02:52:50.232177 14819 tablet_replica.cc:442] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:52:50.232764 14831 tablet_replica.cc:439] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf: TxnStatusTablet state changed. Reason: New leader 0b1aebda96e6457392f3312ebba8cc04. Latest consensus state: current_term: 1 leader_uuid: "0b1aebda96e6457392f3312ebba8cc04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } } }
I20260925 02:52:50.232869 14831 tablet_replica.cc:442] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:52:50.234596 14820 tablet_replica.cc:439] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b1aebda96e6457392f3312ebba8cc04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } } }
I20260925 02:52:50.234685 14820 tablet_replica.cc:442] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:52:50.234799 14820 tablet_replica.cc:439] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b1aebda96e6457392f3312ebba8cc04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } } }
I20260925 02:52:50.234869 14820 tablet_replica.cc:442] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 0b1aebda96e6457392f3312ebba8cc04: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:52:50.235009 14831 tablet_replica.cc:439] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b1aebda96e6457392f3312ebba8cc04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } } }
I20260925 02:52:50.235095 14831 tablet_replica.cc:442] T a9595d4f7b7a4a2b8f2a7313456b5b7b P 671f35701ace4ac2a5e0dc4475aeebbf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:52:50.235167 14864 mvcc.cc:205] Tried to move back new op lower bound from 7333088338857951232 to 7333088338757763072. Current Snapshot: MvccSnapshot[applied={T|T < 7333088338857951232}]
I20260925 02:52:50.237112 14819 tablet_replica.cc:439] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b1aebda96e6457392f3312ebba8cc04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 } } peers { permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 } } peers { permanent_uuid: "0b1aebda96e6457392f3312ebba8cc04" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45137 } } }
I20260925 02:52:50.237211 14819 tablet_replica.cc:442] T a9595d4f7b7a4a2b8f2a7313456b5b7b P edfff45819664869907c7eeeccdd3ac5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:52:50.245529 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:50.242536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54942 (local address 127.12.222.195:45137)
0925 02:52:50.242649 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:50.242653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:50.242781 (+ 128us) server_negotiation.cc:407] Connection header received
0925 02:52:50.242924 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:50.242927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:50.242966 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:50.243045 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:50.243287 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.243731 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.244607 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.244805 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.244916 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:50.245144 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:50.245209 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:50.245276 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:50.245358 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 02:52:50.245399 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:52:50.254441 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:50.251570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:47541 (local address 127.12.222.195:45137)
0925 02:52:50.251652 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:50.251656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:50.251732 (+ 76us) server_negotiation.cc:407] Connection header received
0925 02:52:50.251853 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:50.251855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:50.251887 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:50.251956 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:50.252165 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.252758 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.253602 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.253826 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.253871 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:50.254104 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:50.254164 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:50.254229 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:50.254302 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 02:52:50.254335 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":32}
I20260925 02:52:50.254632 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:50.251358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:45137 (local address 127.12.222.195:47541)
0925 02:52:50.251669 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:50.251682 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:50.251774 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:50.251980 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:50.251994 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:50.251999 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:52:50.252076 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:50.252081 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.252850 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.252855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:50.253481 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:50.253491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.253929 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.253930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:50.253977 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:50.254004 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:52:50.254231 (+ 227us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:52:50.254238 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:52:50.254287 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 02:52:50.254349 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260925 02:52:50.259358 14602 raft_consensus.cc:1261] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf [term 1 FOLLOWER]: Refusing update from remote peer 0b1aebda96e6457392f3312ebba8cc04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:50.259610 14461 raft_consensus.cc:1261] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5 [term 1 FOLLOWER]: Refusing update from remote peer 0b1aebda96e6457392f3312ebba8cc04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:50.259789 14820 consensus_queue.cc:1035] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "671f35701ace4ac2a5e0dc4475aeebbf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:50.259979 14844 consensus_queue.cc:1035] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edfff45819664869907c7eeeccdd3ac5" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:50.295550 14301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:52:50.295663 14301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:52:50.297443 14301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 505a116f63b24500a4b5eeb9fe873447: Sending DeleteTablet for 3 replicas of tablet 4af75471fb7b4312a1b3313d76a030c7
I20260925 02:52:50.297942 14723 tablet_service.cc:1514] Processing DeleteTablet for tablet 4af75471fb7b4312a1b3313d76a030c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:50 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:54920
I20260925 02:52:50.298067 14441 tablet_service.cc:1514] Processing DeleteTablet for tablet 4af75471fb7b4312a1b3313d76a030c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:50 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:52908
I20260925 02:52:50.298296 14582 tablet_service.cc:1514] Processing DeleteTablet for tablet 4af75471fb7b4312a1b3313d76a030c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:50 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:43020
I20260925 02:52:50.298332 14880 tablet_replica.cc:330] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5: stopping tablet replica
I20260925 02:52:50.298451 14880 raft_consensus.cc:2229] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:52:50.298555 14880 raft_consensus.cc:2258] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:50.299280 14880 ts_tablet_manager.cc:1905] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:50.299314 14879 tablet_replica.cc:330] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04: stopping tablet replica
I20260925 02:52:50.299407 14879 raft_consensus.cc:2229] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:52:50.299645 14879 raft_consensus.cc:2258] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:50.300330 14879 ts_tablet_manager.cc:1905] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:50.302589 14880 ts_tablet_manager.cc:1918] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:52:50.302668 14880 log.cc:1196] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-0/wal/wals/4af75471fb7b4312a1b3313d76a030c7
I20260925 02:52:50.302920 14880 ts_tablet_manager.cc:1939] T 4af75471fb7b4312a1b3313d76a030c7 P edfff45819664869907c7eeeccdd3ac5: Deleting consensus metadata
I20260925 02:52:50.303103 14881 tablet_replica.cc:330] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf: stopping tablet replica
I20260925 02:52:50.303215 14881 raft_consensus.cc:2229] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:52:50.303333 14881 raft_consensus.cc:2258] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:50.303346 14879 ts_tablet_manager.cc:1918] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:52:50.303483 14879 log.cc:1196] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/4af75471fb7b4312a1b3313d76a030c7
I20260925 02:52:50.303776 14879 ts_tablet_manager.cc:1939] T 4af75471fb7b4312a1b3313d76a030c7 P 0b1aebda96e6457392f3312ebba8cc04: Deleting consensus metadata
I20260925 02:52:50.303923 14881 ts_tablet_manager.cc:1905] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:50.304352 14287 catalog_manager.cc:4841] TS edfff45819664869907c7eeeccdd3ac5 (127.12.222.193:42091): tablet 4af75471fb7b4312a1b3313d76a030c7 (table test-table [id=3a832ec0ba04472b8494b9f7a68f981a]) successfully deleted
I20260925 02:52:50.304849 14286 catalog_manager.cc:4841] TS 0b1aebda96e6457392f3312ebba8cc04 (127.12.222.195:45137): tablet 4af75471fb7b4312a1b3313d76a030c7 (table test-table [id=3a832ec0ba04472b8494b9f7a68f981a]) successfully deleted
I20260925 02:52:50.307253 14881 ts_tablet_manager.cc:1918] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:52:50.307325 14881 log.cc:1196] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/4af75471fb7b4312a1b3313d76a030c7
I20260925 02:52:50.307518 14881 ts_tablet_manager.cc:1939] T 4af75471fb7b4312a1b3313d76a030c7 P 671f35701ace4ac2a5e0dc4475aeebbf: Deleting consensus metadata
Sep 25 02:52:50 dist-test-slave-252r krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304769, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.222.193@KRBTEST.COM
I20260925 02:52:50.308248 14287 catalog_manager.cc:4841] TS 671f35701ace4ac2a5e0dc4475aeebbf (127.12.222.194:38737): tablet 4af75471fb7b4312a1b3313d76a030c7 (table test-table [id=3a832ec0ba04472b8494b9f7a68f981a]) successfully deleted
I20260925 02:52:50.311702 14810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:50.299069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.12.222.193:42091)
0925 02:52:50.299175 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:50.299179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:50.300025 (+ 846us) server_negotiation.cc:407] Connection header received
0925 02:52:50.300190 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:50.300193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:50.300265 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:50.300315 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:50.301366 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.302114 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.303339 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.303710 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.303786 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:50.308318 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:50.308330 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:50.308346 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:50.308370 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:50.310176 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:50.310551 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:50.310557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:50.310559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:50.310645 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:50.310876 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:50.310879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:50.310881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:50.311145 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:50.311347 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:50.311433 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 02:52:50.311542 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 02:52:50.312103 14501 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52918
I20260925 02:52:50.320729 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:50.313774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.12.222.254:33123)
0925 02:52:50.313885 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:50.313888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:50.314067 (+ 179us) server_negotiation.cc:407] Connection header received
0925 02:52:50.314196 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:50.314199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:50.314243 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:50.314311 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:50.314857 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.315336 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.316045 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:50.316368 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:50.316477 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:50.317717 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:50.317721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:50.317724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:50.317748 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:50.319098 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:50.319495 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:50.319498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:50.319500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:50.319583 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:50.319866 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:50.319869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:50.319871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:50.320071 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:50.320190 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:50.320445 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 02:52:50.320574 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 02:52:50.321080 14301 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53722
I20260925 02:52:50.322281 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 14376
I20260925 02:52:50.331140 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 14516
I20260925 02:52:50.338002 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 14658
I20260925 02:52:50.343515 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 14270
2026-09-25T02:52:50Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2712 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:50 dist-test-slave-252r krb5kdc[14888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:50 dist-test-slave-252r krb5kdc[14888](info): set up 2 sockets
Sep 25 02:52:50 dist-test-slave-252r krb5kdc[14888](info): commencing operation
krb5kdc: starting...
W20260925 02:52:51.307798 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:52:51 dist-test-slave-252r krb5kdc[14888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:52:51.419275 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:38297
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:42945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:38297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:51.517038 14901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:51.517270 14901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:51.517342 14901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:51.520967 14901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:51.521039 14901 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:52:51.521078 14901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:51.521137 14901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:51.521175 14901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:51.521209 14901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:51.525902 14901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:42945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:38297
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:38297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:51.526701 14901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:51.527540 14901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:51.535072 14906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:51.535077 14910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:51.535168 14908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:51.535435 14901 server_base.cc:1034] running on GCE node
I20260925 02:52:51.535909 14901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:51.536919 14901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:51.538071 14901 hybrid_clock.cc:648] HybridClock initialized: now 1790304771538047 us; error 49 us; skew 500 ppm
Sep 25 02:52:51 dist-test-slave-252r krb5kdc[14888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304771, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:51.540410 14901 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:52:51.541178 14901 webserver.cc:460] Webserver started at http://127.12.222.254:40189/ using document root <none> and password file <none>
I20260925 02:52:51.541640 14901 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:51.541721 14901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:51.541917 14901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:51.543690 14901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "ee398adc75504eb69e9898e86446a9a3"
format_stamp: "Formatted at 2026-09-25 02:52:51 on dist-test-slave-252r"
server_key: "6efbdbade17cc48f2cc492bf93b8eac9"
server_key_iv: "8ba1f97d7eb9dc634ff8ed042410466f"
server_key_version: "encryptionkey@0"
I20260925 02:52:51.544193 14901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "ee398adc75504eb69e9898e86446a9a3"
format_stamp: "Formatted at 2026-09-25 02:52:51 on dist-test-slave-252r"
server_key: "6efbdbade17cc48f2cc492bf93b8eac9"
server_key_iv: "8ba1f97d7eb9dc634ff8ed042410466f"
server_key_version: "encryptionkey@0"
I20260925 02:52:51.547384 14901 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:52:51.549424 14917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:51.550470 14901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:52:51.550623 14901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "ee398adc75504eb69e9898e86446a9a3"
format_stamp: "Formatted at 2026-09-25 02:52:51 on dist-test-slave-252r"
server_key: "6efbdbade17cc48f2cc492bf93b8eac9"
server_key_iv: "8ba1f97d7eb9dc634ff8ed042410466f"
server_key_version: "encryptionkey@0"
I20260925 02:52:51.550733 14901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:51.563596 14901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:51.565990 14901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:51.566187 14901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:51.572973 14901 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:38297
I20260925 02:52:51.573009 14979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:38297 every 8 connection(s)
I20260925 02:52:51.573793 14901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:51.576045 14980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:51.581415 14980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3: Bootstrap starting.
I20260925 02:52:51.583657 14980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:51.583685 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 14901
I20260925 02:52:51.583762 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:52:51.584024 13179 external_mini_cluster.cc:1428] Setting key 44d1f187cb56eea506eeb895b992c0e3
I20260925 02:52:51.584481 14980 log.cc:824] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:51.586663 14980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3: No bootstrap required, opened a new log
I20260925 02:52:51.589551 14980 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee398adc75504eb69e9898e86446a9a3" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38297 } }
I20260925 02:52:51.589782 14980 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:51.589823 14980 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee398adc75504eb69e9898e86446a9a3, State: Initialized, Role: FOLLOWER
I20260925 02:52:51.590190 14980 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee398adc75504eb69e9898e86446a9a3" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38297 } }
I20260925 02:52:51.590301 14980 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:51.590379 14980 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:51.590485 14980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:51.591553 14980 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee398adc75504eb69e9898e86446a9a3" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38297 } }
I20260925 02:52:51.591951 14980 leader_election.cc:302] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee398adc75504eb69e9898e86446a9a3; no voters:
I20260925 02:52:51.592273 14980 leader_election.cc:288] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:51.592388 14985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:51.592602 14985 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3 [term 1 LEADER]: Becoming Leader. State: Replica: ee398adc75504eb69e9898e86446a9a3, State: Running, Role: LEADER
Sep 25 02:52:51 dist-test-slave-252r krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:51.592885 14985 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee398adc75504eb69e9898e86446a9a3" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38297 } }
I20260925 02:52:51.593531 14980 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:51.594666 14986 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee398adc75504eb69e9898e86446a9a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee398adc75504eb69e9898e86446a9a3" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38297 } } }
I20260925 02:52:51.594796 14986 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:51.595012 14987 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee398adc75504eb69e9898e86446a9a3. Latest consensus state: current_term: 1 leader_uuid: "ee398adc75504eb69e9898e86446a9a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee398adc75504eb69e9898e86446a9a3" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38297 } } }
I20260925 02:52:51.595116 14987 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3 [sys.catalog]: This master's current role is: LEADER
W20260925 02:52:51.598308 15000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:52:51.598379 15000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:52:51.598445 14994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:51.599211 14994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:51.599788 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:51.585039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37190 (local address 127.12.222.254:38297)
0925 02:52:51.585462 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:51.585476 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:51.585506 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:52:51.586109 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:51.586125 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:51.586387 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:51.586623 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:51.587435 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:51.588965 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:51.590028 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:51.590666 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:51.590777 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:51.593561 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:51.593566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:51.593580 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:51.593606 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:51.596574 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:51.597049 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:51.597054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:51.597059 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:51.597179 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:51.597451 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:51.597454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:51.597456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:51.597900 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:51.598080 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:51.598686 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 02:52:51.598984 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":144,"threads_started":1}
I20260925 02:52:51.604205 14994 catalog_manager.cc:1347] Generated new cluster ID: f5e1eee9da1441aab4df3a264c46ffae
I20260925 02:52:51.604277 14994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:51.622028 14994 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:51.623064 14994 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:51.633297 14994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3: Generated new TSK 0
I20260925 02:52:51.633903 14994 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
I20260925 02:52:51.658553 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38297
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:42945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:51.763062 15006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:51.763269 15006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:51.763360 15006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:51.767050 15006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:51.767089 15006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:51.767143 15006 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:52:51.767162 15006 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:52:51.767191 15006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:52:51.771752 15006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:42945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.222.254:38297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:51.772557 15006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:51.773371 15006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:51.781733 15014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:51.781739 15012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:51.781772 15011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:51.781867 15006 server_base.cc:1034] running on GCE node
I20260925 02:52:51.782338 15006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:51.782902 15006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:51.784067 15006 hybrid_clock.cc:648] HybridClock initialized: now 1790304771784056 us; error 23 us; skew 500 ppm
Sep 25 02:52:51 dist-test-slave-252r krb5kdc[14888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304771, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:51.786497 15006 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:52:51.787142 15006 webserver.cc:460] Webserver started at http://127.12.222.193:36235/ using document root <none> and password file <none>
I20260925 02:52:51.787565 15006 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:51.787616 15006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:51.787768 15006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:51.789553 15006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "ab6d789eaaf140ec8119a3912c9e31ab"
format_stamp: "Formatted at 2026-09-25 02:52:51 on dist-test-slave-252r"
server_key: "f73a876406592208688127878545a5ff"
server_key_iv: "239ded6d368d58becba15490ebe23d2e"
server_key_version: "encryptionkey@0"
I20260925 02:52:51.790001 15006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "ab6d789eaaf140ec8119a3912c9e31ab"
format_stamp: "Formatted at 2026-09-25 02:52:51 on dist-test-slave-252r"
server_key: "f73a876406592208688127878545a5ff"
server_key_iv: "239ded6d368d58becba15490ebe23d2e"
server_key_version: "encryptionkey@0"
I20260925 02:52:51.793351 15006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:52:51.795728 15021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:51.796933 15006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:52:51.797070 15006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "ab6d789eaaf140ec8119a3912c9e31ab"
format_stamp: "Formatted at 2026-09-25 02:52:51 on dist-test-slave-252r"
server_key: "f73a876406592208688127878545a5ff"
server_key_iv: "239ded6d368d58becba15490ebe23d2e"
server_key_version: "encryptionkey@0"
I20260925 02:52:51.797178 15006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:51.841777 15006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:51.844450 15006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:51.844627 15006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:51.845881 15006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:51.845956 15006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:51.846042 15006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:51.846094 15006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:52:51 dist-test-slave-252r krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304771, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:51.855969 15006 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:43777
I20260925 02:52:51.856062 15137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:43777 every 8 connection(s)
I20260925 02:52:51.856832 15006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:52:51.859200 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:51.846740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:51959 (local address 127.12.222.254:38297)
0925 02:52:51.846889 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:51.846894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:51.847694 (+ 800us) server_negotiation.cc:407] Connection header received
0925 02:52:51.848865 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:51.848871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:51.848946 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:51.849045 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:51.850473 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:51.851333 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:51.853124 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:51.853487 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:51.853602 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:51.856184 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:51.856187 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:51.856190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:51.856215 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:51.857748 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:51.858139 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:51.858141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:51.858143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:51.858207 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:51.858426 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:51.858428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:51.858429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:51.858579 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:51.858647 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:51.859014 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 02:52:51.859095 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:52:51.859899 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:51.847107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38297 (local address 127.12.222.193:51959)
0925 02:52:51.847539 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:51.847585 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:51.848533 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:51.849248 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:51.849258 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:51.849681 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:51.850301 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:51.850321 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:51.851467 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:51.851472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:51.852995 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:51.853007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:51.853637 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:51.853640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:51.853769 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:51.854287 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:51.854307 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:51.855990 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:51.857830 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:51.857834 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:51.857842 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:51.858059 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:51.858259 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:51.858261 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:51.858262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:51.858359 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:51.858662 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:51.858666 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:51.858835 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 02:52:51.858970 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 02:52:51.859155 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":122,"threads_started":1}
I20260925 02:52:51.861145 15138 heartbeater.cc:347] Connected to a master server at 127.12.222.254:38297
I20260925 02:52:51.861287 15138 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:51.861725 15138 heartbeater.cc:511] Master 127.12.222.254:38297 requested a full tablet report, sending...
I20260925 02:52:51.863174 14933 ts_manager.cc:194] Registered new tserver with Master: ab6d789eaaf140ec8119a3912c9e31ab (127.12.222.193:43777)
I20260925 02:52:51.864328 14933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:51959
I20260925 02:52:51.864826 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 15006
I20260925 02:52:51.864957 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:52:51.865269 13179 external_mini_cluster.cc:1428] Setting key dd10ad4e2c73082242ab0dadaf6f8fd5
I20260925 02:52:51.870262 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:51.863075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:43989 (local address 127.12.222.254:38297)
0925 02:52:51.863190 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:51.863194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:51.863318 (+ 124us) server_negotiation.cc:407] Connection header received
0925 02:52:51.863507 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:51.863510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:51.863560 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:51.863648 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:51.864043 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:51.864662 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:51.865723 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:51.866006 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:51.866108 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:51.867128 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:51.867134 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:51.867140 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:51.867168 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:51.868771 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:51.869235 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:51.869237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:51.869238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:51.869303 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:51.869590 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:51.869592 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:51.869593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:51.869728 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:51.869798 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:51.870094 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 02:52:51.870168 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:52:51.870430 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:51.862992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38297 (local address 127.12.222.193:43989)
0925 02:52:51.863243 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:51.863256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:51.863378 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:51.863663 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:51.863666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:51.863842 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:51.863925 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:51.863930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:51.864790 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:51.864798 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:51.865600 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:51.865609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:51.866126 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:51.866129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:51.866218 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:51.866676 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:51.866691 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:51.866990 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:51.868873 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:51.868879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:51.868883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:51.869141 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:51.869389 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:51.869393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:51.869397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:51.869501 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:51.869820 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:51.869825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:51.869928 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 02:52:51.870048 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 02:52:51.870196 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":195,"threads_started":1}
WARNING: no policy specified for kudu/127.12.222.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
I20260925 02:52:51.890991 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38297
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:42945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:51.991852 15147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:51.992091 15147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:51.992167 15147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:51.996078 15147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:51.996155 15147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:51.996222 15147 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:52:51.996289 15147 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:52:51.996363 15147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:52:52.001179 15147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:42945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.222.254:38297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:52.002009 15147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:52.002884 15147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:52.011369 15155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:52.011485 15147 server_base.cc:1034] running on GCE node
W20260925 02:52:52.011349 15152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:52.011385 15153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:52.011992 15147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:52.012637 15147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:52.013785 15147 hybrid_clock.cc:648] HybridClock initialized: now 1790304772013759 us; error 49 us; skew 500 ppm
Sep 25 02:52:52 dist-test-slave-252r krb5kdc[14888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304772, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:52.016211 15147 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:52:52.017062 15147 webserver.cc:460] Webserver started at http://127.12.222.194:32935/ using document root <none> and password file <none>
I20260925 02:52:52.017536 15147 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:52.017589 15147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:52.017789 15147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:52.019526 15147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "630879a0cbdc4d9ba586b0bc1439c387"
format_stamp: "Formatted at 2026-09-25 02:52:52 on dist-test-slave-252r"
server_key: "dcb6be45b4d569a5d960caca1c1e9f0e"
server_key_iv: "125e38bc1728b30884a23173c2d5b5b4"
server_key_version: "encryptionkey@0"
I20260925 02:52:52.020032 15147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "630879a0cbdc4d9ba586b0bc1439c387"
format_stamp: "Formatted at 2026-09-25 02:52:52 on dist-test-slave-252r"
server_key: "dcb6be45b4d569a5d960caca1c1e9f0e"
server_key_iv: "125e38bc1728b30884a23173c2d5b5b4"
server_key_version: "encryptionkey@0"
I20260925 02:52:52.023343 15147 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260925 02:52:52.025416 15162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:52.026441 15147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:52:52.026551 15147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "630879a0cbdc4d9ba586b0bc1439c387"
format_stamp: "Formatted at 2026-09-25 02:52:52 on dist-test-slave-252r"
server_key: "dcb6be45b4d569a5d960caca1c1e9f0e"
server_key_iv: "125e38bc1728b30884a23173c2d5b5b4"
server_key_version: "encryptionkey@0"
I20260925 02:52:52.026640 15147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:52.034564 15147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:52.036944 15147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:52.037125 15147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:52.038311 15147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:52.038357 15147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:52.038415 15147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:52.038443 15147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:52:52 dist-test-slave-252r krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304772, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:52.048970 15147 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:36749
I20260925 02:52:52.049101 15278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:36749 every 8 connection(s)
I20260925 02:52:52.049911 15147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:52:52.051131 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.038960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:49945 (local address 127.12.222.254:38297)
0925 02:52:52.039072 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.039076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.039994 (+ 918us) server_negotiation.cc:407] Connection header received
0925 02:52:52.040879 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.040883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.040948 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.041049 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:52.042269 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.043012 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.044204 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.044452 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.044554 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.047622 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:52.047626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:52.047640 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:52.047670 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:52.049203 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:52.049677 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:52.049680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:52.049682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:52.049768 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:52.050081 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:52.050083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:52.050084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:52.050235 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:52.050337 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.050836 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.050956 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:52:52.051739 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.039413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38297 (local address 127.12.222.194:49945)
0925 02:52:52.039863 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.039900 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.040667 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.041208 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.041216 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.041641 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:52.042138 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.042150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.043127 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.043130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.044086 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.044096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.044595 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.044598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.044753 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.045252 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:52.045268 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:52.047442 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:52.049302 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:52.049306 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:52.049315 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:52.049577 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:52.049847 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:52.049849 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:52.049850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:52.049951 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:52.050337 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:52.050344 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:52.050619 (+ 275us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.050795 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.051082 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":128,"threads_started":1}
I20260925 02:52:52.052906 15279 heartbeater.cc:347] Connected to a master server at 127.12.222.254:38297
I20260925 02:52:52.053046 15279 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:52.053493 15279 heartbeater.cc:511] Master 127.12.222.254:38297 requested a full tablet report, sending...
I20260925 02:52:52.054917 14934 ts_manager.cc:194] Registered new tserver with Master: 630879a0cbdc4d9ba586b0bc1439c387 (127.12.222.194:36749)
I20260925 02:52:52.055441 14934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:49945
I20260925 02:52:52.055649 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 15147
I20260925 02:52:52.055749 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:52:52.056026 13179 external_mini_cluster.cc:1428] Setting key f69c946f9eff438ff34ae0e03634b524
I20260925 02:52:52.061559 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.054727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:43623 (local address 127.12.222.254:38297)
0925 02:52:52.054829 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.054832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.054983 (+ 151us) server_negotiation.cc:407] Connection header received
0925 02:52:52.055089 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.055092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.055128 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.055210 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:52.055647 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.056196 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.057384 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.057577 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.057672 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.058625 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:52.058628 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:52.058630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:52.058656 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:52.059928 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:52.060353 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:52.060356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:52.060358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:52.060427 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:52.060687 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:52.060690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:52.060692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:52.060906 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:52.061011 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.061307 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.061407 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:52:52.061571 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.054687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38297 (local address 127.12.222.194:43623)
0925 02:52:52.054894 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.054906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.055011 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.055222 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.055225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.055392 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:52.055533 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.055542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.056385 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.056388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.057255 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.057267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.057685 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.057688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.057772 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.058204 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:52.058214 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:52.058479 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:52.060045 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:52.060047 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:52.060049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:52.060273 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:52.060513 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:52.060517 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:52.060519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:52.060603 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:52.061003 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:52.061006 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:52.061105 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.061252 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.061375 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.12.222.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
I20260925 02:52:52.082543 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38297
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:42945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:52.185238 15288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:52.185420 15288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:52.185458 15288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:52.189235 15288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:52.189280 15288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:52.189350 15288 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:52:52.189397 15288 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:52:52.189455 15288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:52:52.194173 15288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:42945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.222.254:38297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:52.195071 15288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:52.195950 15288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:52.204419 15294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:52.204438 15293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:52.204464 15296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:52.204483 15288 server_base.cc:1034] running on GCE node
I20260925 02:52:52.204994 15288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:52.205515 15288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:52.206688 15288 hybrid_clock.cc:648] HybridClock initialized: now 1790304772206673 us; error 27 us; skew 500 ppm
Sep 25 02:52:52 dist-test-slave-252r krb5kdc[14888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304772, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:52.209056 15288 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:52:52.209699 15288 webserver.cc:460] Webserver started at http://127.12.222.195:36793/ using document root <none> and password file <none>
I20260925 02:52:52.210124 15288 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:52.210171 15288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:52.210299 15288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:52.211877 15288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "2a43a93680db4067a02fc6016c0e2a0e"
format_stamp: "Formatted at 2026-09-25 02:52:52 on dist-test-slave-252r"
server_key: "e36880dae6ce1816d1b9f4533fd53bdb"
server_key_iv: "45316c952aae11f533fbcf1de3f03e6e"
server_key_version: "encryptionkey@0"
I20260925 02:52:52.212406 15288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "2a43a93680db4067a02fc6016c0e2a0e"
format_stamp: "Formatted at 2026-09-25 02:52:52 on dist-test-slave-252r"
server_key: "e36880dae6ce1816d1b9f4533fd53bdb"
server_key_iv: "45316c952aae11f533fbcf1de3f03e6e"
server_key_version: "encryptionkey@0"
I20260925 02:52:52.215405 15288 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:52:52.217428 15303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:52.218427 15288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:52:52.218508 15288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "2a43a93680db4067a02fc6016c0e2a0e"
format_stamp: "Formatted at 2026-09-25 02:52:52 on dist-test-slave-252r"
server_key: "e36880dae6ce1816d1b9f4533fd53bdb"
server_key_iv: "45316c952aae11f533fbcf1de3f03e6e"
server_key_version: "encryptionkey@0"
I20260925 02:52:52.218577 15288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:52.237174 15288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:52.240316 15288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:52.240454 15288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:52.241676 15288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:52.241724 15288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:52.241760 15288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:52.241825 15288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:52:52 dist-test-slave-252r krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304772, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:52.252532 15288 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:35437
I20260925 02:52:52.252620 15419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:35437 every 8 connection(s)
I20260925 02:52:52.253547 15288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:52:52.254717 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.242647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:38473 (local address 127.12.222.254:38297)
0925 02:52:52.242804 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.242809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.243493 (+ 684us) server_negotiation.cc:407] Connection header received
0925 02:52:52.244604 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.244607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.244664 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.244749 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:52.246307 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.247084 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.248178 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.248494 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.248595 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.250995 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:52.251001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:52.251003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:52.251032 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:52.252645 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:52.253228 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:52.253231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:52.253232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:52.253305 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:52.253709 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:52.253712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:52.253713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:52.253868 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:52.254002 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.254526 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.254612 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:52:52.255468 15312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.242914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38297 (local address 127.12.222.195:38473)
0925 02:52:52.243327 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.243374 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.244405 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.244987 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.244999 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.245516 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:52.246144 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.246164 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.247205 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.247223 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.248051 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.248060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.248656 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.248660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.248805 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.249331 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:52.249356 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:52.250842 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:52.252777 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:52.252783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:52.252794 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:52.253130 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:52.253400 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:52.253403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:52.253405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:52.253572 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:52.253992 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:52.254000 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:52.254251 (+ 251us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.254464 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.254663 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":119,"threads_started":1}
I20260925 02:52:52.256554 15420 heartbeater.cc:347] Connected to a master server at 127.12.222.254:38297
I20260925 02:52:52.256704 15420 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:52.257136 15420 heartbeater.cc:511] Master 127.12.222.254:38297 requested a full tablet report, sending...
I20260925 02:52:52.257357 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 15288
I20260925 02:52:52.257480 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:52:52.257778 13179 external_mini_cluster.cc:1428] Setting key c942aaf0cce4323cfb93de7915ff11f1
I20260925 02:52:52.258149 14934 ts_manager.cc:194] Registered new tserver with Master: 2a43a93680db4067a02fc6016c0e2a0e (127.12.222.195:35437)
I20260925 02:52:52.258880 14934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:38473
I20260925 02:52:52.259851 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:52:52.266851 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.258736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:41453 (local address 127.12.222.254:38297)
0925 02:52:52.258853 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.258857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.259136 (+ 279us) server_negotiation.cc:407] Connection header received
0925 02:52:52.259223 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.259225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.259267 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.259345 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:52.259849 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.260605 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.261923 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.262288 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.262410 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.263393 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:52.263400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:52.263406 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:52.263435 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:52.265143 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:52.265661 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:52.265663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:52.265664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:52.265730 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:52.266033 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:52.266039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:52.266043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:52.266237 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:52.266322 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.266645 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.266756 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:52:52.266955 15426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.258666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38297 (local address 127.12.222.195:41453)
0925 02:52:52.259040 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.259055 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.259149 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.259382 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.259385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.259582 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:52.259728 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.259738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.260755 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.260762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.261796 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.261811 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.262438 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.262441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.262525 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.262974 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:52.262984 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:52.263249 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:52.265253 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:52.265255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:52.265256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:52.265557 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:52.265825 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:52.265830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:52.265835 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:52.265934 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:52.266346 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:52.266352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:52.266463 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.266578 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.266731 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
Sep 25 02:52:52 dist-test-slave-252r krb5kdc[14888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:52:52 dist-test-slave-252r krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:52.285118 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.277969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.12.222.254:38297)
0925 02:52:52.278094 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.278097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.278108 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:52:52.278139 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.278141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.278173 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.278237 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:52.278860 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.279318 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.280092 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.280338 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.280451 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.282630 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:52.282633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:52.282635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:52.282655 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:52.283803 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:52.284155 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:52.284158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:52.284159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:52.284223 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:52.284518 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:52.284520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:52.284521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:52.284666 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:52.284762 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.284935 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.285021 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:52:52.287626 14933 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:52:52.290372 14933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:52:52.304705 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.299007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56110 (local address 127.12.222.194:36749)
0925 02:52:52.299621 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.299631 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.299848 (+ 217us) server_negotiation.cc:407] Connection header received
0925 02:52:52.300052 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.300059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.300246 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.300413 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:52.300652 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.301730 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.302999 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.303808 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.303982 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.304108 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.304474 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.304564 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":105,"threads_started":1}
I20260925 02:52:52.304731 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.298201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:35437 (local address 127.0.0.1:55092)
0925 02:52:52.299167 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.299218 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.299310 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.299858 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.299866 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.299891 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:52.300044 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.300049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.301553 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.301557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.303014 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.303026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.303961 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.303964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.304223 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.304315 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.304428 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.304575 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":202,"thread_start_us":70,"threads_started":1}
I20260925 02:52:52.304733 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.298619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:36749 (local address 127.0.0.1:56110)
0925 02:52:52.299768 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.299784 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.299903 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.300415 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.300417 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.300434 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:52.300541 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.300547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.301830 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.301834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.302875 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.302885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.303952 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.303956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.304219 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.304321 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.304428 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.304575 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"mutex_wait_us":85,"thread_start_us":92,"threads_started":1}
I20260925 02:52:52.305322 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.298344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.12.222.195:35437)
0925 02:52:52.299252 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.299265 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.299288 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:52:52.299468 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.299474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.299682 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.300213 (+ 531us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:52.300422 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.301399 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.303137 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.303855 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.303994 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.304108 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.304948 (+ 840us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.305190 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":109,"threads_started":1}
I20260925 02:52:52.307685 15354 tablet_service.cc:1467] Processing CreateTablet for tablet a84af9354b204435b6c9d4caa264d1e4 (DEFAULT_TABLE table=test-table [id=95c64a9954e34ba183e4a6ffb20eda25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:52.308733 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.298399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:43777 (local address 127.0.0.1:33178)
0925 02:52:52.299473 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.299487 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.299571 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.300118 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.300121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.300144 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:52.300255 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.300263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.301702 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.301705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.303014 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.303026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.305135 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.305141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.308420 (+ 3279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.308456 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.308539 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.308606 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"mutex_wait_us":41,"thread_start_us":69,"threads_started":1}
I20260925 02:52:52.308741 15354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a84af9354b204435b6c9d4caa264d1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:52.308817 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.298838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33178 (local address 127.12.222.193:43777)
0925 02:52:52.299143 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.299157 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.299666 (+ 509us) server_negotiation.cc:407] Connection header received
0925 02:52:52.299751 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.299758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.299979 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.300165 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:52.300374 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.301597 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.304167 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.304998 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.305151 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.305266 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.308595 (+ 3329us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.308685 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":91,"threads_started":1}
I20260925 02:52:52.310130 15213 tablet_service.cc:1467] Processing CreateTablet for tablet a84af9354b204435b6c9d4caa264d1e4 (DEFAULT_TABLE table=test-table [id=95c64a9954e34ba183e4a6ffb20eda25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:52.310900 15213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a84af9354b204435b6c9d4caa264d1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:52.311058 15071 tablet_service.cc:1467] Processing CreateTablet for tablet a84af9354b204435b6c9d4caa264d1e4 (DEFAULT_TABLE table=test-table [id=95c64a9954e34ba183e4a6ffb20eda25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:52.311715 15071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a84af9354b204435b6c9d4caa264d1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:52.317090 15443 tablet_bootstrap.cc:492] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387: Bootstrap starting.
I20260925 02:52:52.319031 15443 tablet_bootstrap.cc:654] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:52.319181 15444 tablet_bootstrap.cc:492] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab: Bootstrap starting.
I20260925 02:52:52.320325 15445 tablet_bootstrap.cc:492] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e: Bootstrap starting.
I20260925 02:52:52.320690 15443 log.cc:824] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:52.321606 15444 tablet_bootstrap.cc:654] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:52.322345 15445 tablet_bootstrap.cc:654] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:52.322374 15444 log.cc:824] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:52.322891 15443 tablet_bootstrap.cc:492] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387: No bootstrap required, opened a new log
I20260925 02:52:52.323087 15443 ts_tablet_manager.cc:1397] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:52:52.323194 15445 log.cc:824] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:52.325022 15444 tablet_bootstrap.cc:492] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab: No bootstrap required, opened a new log
I20260925 02:52:52.325212 15444 ts_tablet_manager.cc:1397] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:52:52.325418 15445 tablet_bootstrap.cc:492] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e: No bootstrap required, opened a new log
I20260925 02:52:52.325588 15445 ts_tablet_manager.cc:1397] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:52:52.326433 15443 raft_consensus.cc:348] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } }
I20260925 02:52:52.326761 15443 raft_consensus.cc:374] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:52.326825 15443 raft_consensus.cc:729] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 630879a0cbdc4d9ba586b0bc1439c387, State: Initialized, Role: FOLLOWER
I20260925 02:52:52.327225 15443 consensus_queue.cc:260] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } }
I20260925 02:52:52.327797 15279 heartbeater.cc:503] Master 127.12.222.254:38297 was elected leader, sending a full tablet report...
I20260925 02:52:52.328154 15443 ts_tablet_manager.cc:1428] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260925 02:52:52.328934 15444 raft_consensus.cc:348] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } }
I20260925 02:52:52.329267 15444 raft_consensus.cc:374] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:52.329136 15445 raft_consensus.cc:348] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } }
I20260925 02:52:52.329341 15445 raft_consensus.cc:374] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:52.329363 15444 raft_consensus.cc:729] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6d789eaaf140ec8119a3912c9e31ab, State: Initialized, Role: FOLLOWER
I20260925 02:52:52.329402 15445 raft_consensus.cc:729] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a43a93680db4067a02fc6016c0e2a0e, State: Initialized, Role: FOLLOWER
I20260925 02:52:52.329809 15445 consensus_queue.cc:260] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } }
I20260925 02:52:52.329874 15444 consensus_queue.cc:260] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } }
I20260925 02:52:52.330514 15420 heartbeater.cc:503] Master 127.12.222.254:38297 was elected leader, sending a full tablet report...
I20260925 02:52:52.330790 15445 ts_tablet_manager.cc:1428] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 02:52:52.330790 15444 ts_tablet_manager.cc:1428] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 02:52:52.331110 15138 heartbeater.cc:503] Master 127.12.222.254:38297 was elected leader, sending a full tablet report...
W20260925 02:52:52.357966 15139 tablet.cc:2369] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:52:52.447963 15449 raft_consensus.cc:482] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:52:52.448144 15449 raft_consensus.cc:504] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } }
I20260925 02:52:52.449235 15449 leader_election.cc:288] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a43a93680db4067a02fc6016c0e2a0e (127.12.222.195:35437), ab6d789eaaf140ec8119a3912c9e31ab (127.12.222.193:43777)
I20260925 02:52:52.452731 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.449594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:41673 (local address 127.12.222.195:35437)
0925 02:52:52.449710 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.449714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.449728 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:52:52.449807 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.449810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.449860 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.449944 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:52.450223 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.450729 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.451715 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.452103 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.452228 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.452305 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.452529 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.452602 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:52:52.452731 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.449456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:35437 (local address 127.12.222.194:41673)
0925 02:52:52.449609 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.449624 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.449717 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.449964 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.449967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.449986 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:52.450103 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.450109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.450845 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.450850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.451588 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.451596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.452222 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.452252 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.452377 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.452395 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.452511 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.452585 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 02:52:52.453426 15453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.449461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:43777 (local address 127.12.222.194:42959)
0925 02:52:52.449965 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.449977 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.450098 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.450323 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.450325 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.450339 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:52.450763 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.450770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.451459 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.451462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.452177 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.452185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.453007 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.453011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.453106 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.453134 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.453232 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.453290 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":123,"thread_start_us":168,"threads_started":1}
I20260925 02:52:52.453478 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.449578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:42959 (local address 127.12.222.193:43777)
0925 02:52:52.449716 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.449724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.450063 (+ 339us) server_negotiation.cc:407] Connection header received
0925 02:52:52.450186 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.450189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.450234 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.450312 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:52.450898 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.451347 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.452334 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.452845 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.452980 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.453023 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.453271 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.453346 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:52:52.453459 15374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a84af9354b204435b6c9d4caa264d1e4" candidate_uuid: "630879a0cbdc4d9ba586b0bc1439c387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a43a93680db4067a02fc6016c0e2a0e" is_pre_election: true
I20260925 02:52:52.453795 15374 raft_consensus.cc:2454] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 630879a0cbdc4d9ba586b0bc1439c387 in term 0.
I20260925 02:52:52.454205 15091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a84af9354b204435b6c9d4caa264d1e4" candidate_uuid: "630879a0cbdc4d9ba586b0bc1439c387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" is_pre_election: true
I20260925 02:52:52.454321 15163 leader_election.cc:302] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a43a93680db4067a02fc6016c0e2a0e, 630879a0cbdc4d9ba586b0bc1439c387; no voters:
I20260925 02:52:52.454530 15091 raft_consensus.cc:2454] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 630879a0cbdc4d9ba586b0bc1439c387 in term 0.
I20260925 02:52:52.454587 15449 raft_consensus.cc:2781] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:52:52.454741 15449 raft_consensus.cc:482] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:52:52.454793 15449 raft_consensus.cc:3037] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:52.455837 15449 raft_consensus.cc:504] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } }
I20260925 02:52:52.456172 15449 leader_election.cc:288] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [CANDIDATE]: Term 1 election: Requested vote from peers 2a43a93680db4067a02fc6016c0e2a0e (127.12.222.195:35437), ab6d789eaaf140ec8119a3912c9e31ab (127.12.222.193:43777)
I20260925 02:52:52.456439 15374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a84af9354b204435b6c9d4caa264d1e4" candidate_uuid: "630879a0cbdc4d9ba586b0bc1439c387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a43a93680db4067a02fc6016c0e2a0e"
I20260925 02:52:52.456463 15091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a84af9354b204435b6c9d4caa264d1e4" candidate_uuid: "630879a0cbdc4d9ba586b0bc1439c387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6d789eaaf140ec8119a3912c9e31ab"
I20260925 02:52:52.456568 15374 raft_consensus.cc:3037] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:52.456584 15091 raft_consensus.cc:3037] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:52.457571 15091 raft_consensus.cc:2454] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 630879a0cbdc4d9ba586b0bc1439c387 in term 1.
I20260925 02:52:52.457702 15374 raft_consensus.cc:2454] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 630879a0cbdc4d9ba586b0bc1439c387 in term 1.
I20260925 02:52:52.457875 15164 leader_election.cc:302] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 630879a0cbdc4d9ba586b0bc1439c387, ab6d789eaaf140ec8119a3912c9e31ab; no voters:
I20260925 02:52:52.458047 15449 raft_consensus.cc:2781] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:52.458210 15449 raft_consensus.cc:686] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 LEADER]: Becoming Leader. State: Replica: 630879a0cbdc4d9ba586b0bc1439c387, State: Running, Role: LEADER
I20260925 02:52:52.458496 15449 consensus_queue.cc:237] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } }
I20260925 02:52:52.461768 14933 catalog_manager.cc:5495] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 reported cstate change: term changed from 0 to 1, leader changed from <none> to 630879a0cbdc4d9ba586b0bc1439c387 (127.12.222.194). New cstate: current_term: 1 leader_uuid: "630879a0cbdc4d9ba586b0bc1439c387" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } health_report { overall_health: HEALTHY } } }
W20260925 02:52:52.504944 15421 tablet.cc:2369] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 02:52:52 dist-test-slave-252r krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304771, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:52.509042 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.501662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38297 (local address 127.0.0.1:37212)
0925 02:52:52.501921 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.501933 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.502024 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.502251 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.502253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.502641 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:52.502728 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.502733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.503389 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.503391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.503913 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.503919 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.504309 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.504311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.504367 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.504986 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:52.504997 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:52.506334 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:52.507744 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:52.507752 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:52.507756 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:52.507950 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:52.508188 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:52.508190 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:52.508191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:52.508321 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:52.508621 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:52.508625 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:52.508714 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.508796 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.508922 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260925 02:52:52.509106 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.501754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37212 (local address 127.12.222.254:38297)
0925 02:52:52.501910 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.501914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.502001 (+ 87us) server_negotiation.cc:407] Connection header received
0925 02:52:52.502109 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.502111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.502145 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.502204 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:52.502822 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.503305 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.504001 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.504180 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.504304 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.506471 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:52.506474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:52.506476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:52.506496 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:52.507655 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:52.508026 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:52.508029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:52.508030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:52.508092 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:52.508394 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:52.508396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:52.508397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:52.508538 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:52.508606 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.508847 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.508980 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":62}
I20260925 02:52:52.512676 14933 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:37212:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:52:52.513365 14933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:52:52.517868 15071 tablet_service.cc:1467] Processing CreateTablet for tablet 50f5c1d5ffa449b89a882bbb09abe06f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f0be743ef604f3f982f9646a5790a6b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:52:52.518038 15354 tablet_service.cc:1467] Processing CreateTablet for tablet 50f5c1d5ffa449b89a882bbb09abe06f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f0be743ef604f3f982f9646a5790a6b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:52:52.517987 15213 tablet_service.cc:1467] Processing CreateTablet for tablet 50f5c1d5ffa449b89a882bbb09abe06f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f0be743ef604f3f982f9646a5790a6b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:52:52.518500 15071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50f5c1d5ffa449b89a882bbb09abe06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:52.518627 15213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50f5c1d5ffa449b89a882bbb09abe06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:52.518627 15354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50f5c1d5ffa449b89a882bbb09abe06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:52.521212 15444 tablet_bootstrap.cc:492] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab: Bootstrap starting.
I20260925 02:52:52.521374 15443 tablet_bootstrap.cc:492] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387: Bootstrap starting.
I20260925 02:52:52.521374 15445 tablet_bootstrap.cc:492] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e: Bootstrap starting.
I20260925 02:52:52.522125 15444 tablet_bootstrap.cc:654] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:52.522346 15445 tablet_bootstrap.cc:654] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:52.522346 15443 tablet_bootstrap.cc:654] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:52.523336 15444 tablet_bootstrap.cc:492] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab: No bootstrap required, opened a new log
I20260925 02:52:52.523432 15444 ts_tablet_manager.cc:1397] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:52.523794 15445 tablet_bootstrap.cc:492] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e: No bootstrap required, opened a new log
I20260925 02:52:52.523888 15445 ts_tablet_manager.cc:1397] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:52:52.523874 15444 raft_consensus.cc:348] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:52.523993 15444 raft_consensus.cc:374] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:52.524034 15444 raft_consensus.cc:729] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6d789eaaf140ec8119a3912c9e31ab, State: Initialized, Role: FOLLOWER
I20260925 02:52:52.524072 15443 tablet_bootstrap.cc:492] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387: No bootstrap required, opened a new log
I20260925 02:52:52.524147 15443 ts_tablet_manager.cc:1397] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260925 02:52:52.524149 15444 consensus_queue.cc:260] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:52.524339 15445 raft_consensus.cc:348] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:52.524470 15445 raft_consensus.cc:374] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:52.524511 15445 raft_consensus.cc:729] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a43a93680db4067a02fc6016c0e2a0e, State: Initialized, Role: FOLLOWER
I20260925 02:52:52.524560 15444 ts_tablet_manager.cc:1428] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:52:52.524627 15443 raft_consensus.cc:348] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:52.524643 15445 consensus_queue.cc:260] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:52.524726 15443 raft_consensus.cc:374] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:52.524756 15443 raft_consensus.cc:729] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 630879a0cbdc4d9ba586b0bc1439c387, State: Initialized, Role: FOLLOWER
I20260925 02:52:52.524864 15443 consensus_queue.cc:260] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:52.524894 15452 tablet_replica.cc:439] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } }
I20260925 02:52:52.525054 15452 tablet_replica.cc:442] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:52:52.525106 15443 ts_tablet_manager.cc:1428] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:52:52.525126 15449 tablet_replica.cc:439] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } }
I20260925 02:52:52.525235 15449 tablet_replica.cc:442] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:52:52.525606 15451 tablet_replica.cc:439] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } }
I20260925 02:52:52.525746 15451 tablet_replica.cc:442] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:52:52.526268 15445 ts_tablet_manager.cc:1428] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260925 02:52:52.551159 15280 tablet.cc:2369] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:52:52.560770 15452 raft_consensus.cc:482] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:52:52.560889 15452 raft_consensus.cc:504] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:52.561925 15452 leader_election.cc:288] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 630879a0cbdc4d9ba586b0bc1439c387 (127.12.222.194:36749), 2a43a93680db4067a02fc6016c0e2a0e (127.12.222.195:35437)
I20260925 02:52:52.565534 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.562205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:49433 (local address 127.12.222.194:36749)
0925 02:52:52.562331 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.562336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.562525 (+ 189us) server_negotiation.cc:407] Connection header received
0925 02:52:52.562666 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.562669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.562712 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.562786 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:52.563049 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.563529 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.564660 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.565013 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.565135 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.565186 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.565373 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.565438 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:52:52.565548 15465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.562127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:36749 (local address 127.12.222.193:49433)
0925 02:52:52.562439 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.562451 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.562566 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.562797 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.562800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.562815 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:52.562944 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.562950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.563640 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.563645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.564547 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.564556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.565152 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.565155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.565248 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.565293 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.565358 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.565417 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":79,"threads_started":1}
I20260925 02:52:52.565755 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.562233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:43201 (local address 127.12.222.195:35437)
0925 02:52:52.562331 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.562336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.562577 (+ 241us) server_negotiation.cc:407] Connection header received
0925 02:52:52.562696 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.562701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.562741 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.562818 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:52.563427 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.563872 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.564841 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.565188 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.565338 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.565380 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.565551 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.565624 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:52:52.566085 15233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50f5c1d5ffa449b89a882bbb09abe06f" candidate_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "630879a0cbdc4d9ba586b0bc1439c387" is_pre_election: true
I20260925 02:52:52.566273 15233 raft_consensus.cc:2454] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab6d789eaaf140ec8119a3912c9e31ab in term 0.
I20260925 02:52:52.566614 15466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.562127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:35437 (local address 127.12.222.193:43201)
0925 02:52:52.562485 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.562497 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.562612 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.562829 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.562832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.562857 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:52.563307 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.563313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.563979 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.563983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.564728 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.564737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.565337 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.565341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.565425 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.565444 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.565524 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.565887 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":18,"spinlock_wait_cycles":1792,"thread_start_us":63,"threads_started":1}
I20260925 02:52:52.566695 15024 leader_election.cc:302] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 630879a0cbdc4d9ba586b0bc1439c387, ab6d789eaaf140ec8119a3912c9e31ab; no voters:
I20260925 02:52:52.566991 15452 raft_consensus.cc:2781] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:52:52.567060 15452 raft_consensus.cc:482] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:52:52.567137 15452 raft_consensus.cc:3037] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:52.567111 15374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50f5c1d5ffa449b89a882bbb09abe06f" candidate_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a43a93680db4067a02fc6016c0e2a0e" is_pre_election: true
I20260925 02:52:52.567248 15374 raft_consensus.cc:2454] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab6d789eaaf140ec8119a3912c9e31ab in term 0.
I20260925 02:52:52.568074 15452 raft_consensus.cc:504] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:52.568415 15452 leader_election.cc:288] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [CANDIDATE]: Term 1 election: Requested vote from peers 630879a0cbdc4d9ba586b0bc1439c387 (127.12.222.194:36749), 2a43a93680db4067a02fc6016c0e2a0e (127.12.222.195:35437)
I20260925 02:52:52.568707 15233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50f5c1d5ffa449b89a882bbb09abe06f" candidate_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "630879a0cbdc4d9ba586b0bc1439c387"
I20260925 02:52:52.568737 15374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50f5c1d5ffa449b89a882bbb09abe06f" candidate_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a43a93680db4067a02fc6016c0e2a0e"
I20260925 02:52:52.568827 15233 raft_consensus.cc:3037] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:52.568842 15374 raft_consensus.cc:3037] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:52.569610 15233 raft_consensus.cc:2454] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6d789eaaf140ec8119a3912c9e31ab in term 1.
I20260925 02:52:52.569614 15374 raft_consensus.cc:2454] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6d789eaaf140ec8119a3912c9e31ab in term 1.
I20260925 02:52:52.569857 15022 leader_election.cc:302] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a43a93680db4067a02fc6016c0e2a0e, ab6d789eaaf140ec8119a3912c9e31ab; no voters:
I20260925 02:52:52.570057 15452 raft_consensus.cc:2781] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:52.570242 15452 raft_consensus.cc:686] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [term 1 LEADER]: Becoming Leader. State: Replica: ab6d789eaaf140ec8119a3912c9e31ab, State: Running, Role: LEADER
I20260925 02:52:52.570542 15452 consensus_queue.cc:237] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:52.572403 15467 tablet_replica.cc:439] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab: TxnStatusTablet state changed. Reason: New leader ab6d789eaaf140ec8119a3912c9e31ab. Latest consensus state: current_term: 1 leader_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } }
I20260925 02:52:52.572526 15467 tablet_replica.cc:442] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:52:52.573558 14934 catalog_manager.cc:5495] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab reported cstate change: term changed from 0 to 1, leader changed from <none> to ab6d789eaaf140ec8119a3912c9e31ab (127.12.222.193). New cstate: current_term: 1 leader_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } health_report { overall_health: UNKNOWN } } }
I20260925 02:52:52.580370 15469 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:52:52.580540 15469 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:52:52.601428 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.598189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33188 (local address 127.12.222.193:43777)
0925 02:52:52.598315 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.598318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.598328 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:52:52.598398 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.598400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.598443 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.598545 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:52.598791 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.599227 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.599991 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.600258 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.600368 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.600669 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:52.601035 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:52.601154 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.601229 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.601294 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:52:52.601429 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.597794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:43777 (local address 127.0.0.1:33188)
0925 02:52:52.597936 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.597952 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.598055 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.598587 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.598590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.598598 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:52:52.598692 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.598697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.599330 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.599336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.599876 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.599884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.600371 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.600374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.600486 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.600526 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:52:52.601133 (+ 607us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:52:52.601150 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.601219 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.601319 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 02:52:52.605957 15233 raft_consensus.cc:1261] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 FOLLOWER]: Refusing update from remote peer ab6d789eaaf140ec8119a3912c9e31ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:52.605953 15374 raft_consensus.cc:1261] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e [term 1 FOLLOWER]: Refusing update from remote peer ab6d789eaaf140ec8119a3912c9e31ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:52.606495 15467 consensus_queue.cc:1035] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:52.606644 15452 consensus_queue.cc:1035] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:52.609562 15449 tablet_replica.cc:439] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387: TxnStatusTablet state changed. Reason: New leader ab6d789eaaf140ec8119a3912c9e31ab. Latest consensus state: current_term: 1 leader_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } }
I20260925 02:52:52.609694 15449 tablet_replica.cc:442] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:52:52.610570 15451 tablet_replica.cc:439] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e: TxnStatusTablet state changed. Reason: New leader ab6d789eaaf140ec8119a3912c9e31ab. Latest consensus state: current_term: 1 leader_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } }
I20260925 02:52:52.610721 15451 tablet_replica.cc:442] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:52:52.611356 15472 mvcc.cc:205] Tried to move back new op lower bound from 7333088348588077056 to 7333088348451205120. Current Snapshot: MvccSnapshot[applied={T|T < 7333088348588077056}]
I20260925 02:52:52.611537 15452 tablet_replica.cc:439] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } }
I20260925 02:52:52.611626 15452 tablet_replica.cc:442] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:52:52.612845 15449 tablet_replica.cc:439] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } }
I20260925 02:52:52.612946 15449 tablet_replica.cc:442] T 50f5c1d5ffa449b89a882bbb09abe06f P 630879a0cbdc4d9ba586b0bc1439c387: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:52:52.614527 15467 tablet_replica.cc:439] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } }
I20260925 02:52:52.614620 15467 tablet_replica.cc:442] T 50f5c1d5ffa449b89a882bbb09abe06f P ab6d789eaaf140ec8119a3912c9e31ab: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:52:52.615326 15451 tablet_replica.cc:439] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } } }
I20260925 02:52:52.615432 15451 tablet_replica.cc:442] T 50f5c1d5ffa449b89a882bbb09abe06f P 2a43a93680db4067a02fc6016c0e2a0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:52:52.616858 15473 mvcc.cc:205] Tried to move back new op lower bound from 7333088348588077056 to 7333088348451205120. Current Snapshot: MvccSnapshot[applied={T|T < 7333088348588077056}]
I20260925 02:52:52.617179 15474 mvcc.cc:205] Tried to move back new op lower bound from 7333088348588077056 to 7333088348451205120. Current Snapshot: MvccSnapshot[applied={T|T < 7333088348588077056}]
I20260925 02:52:52.623999 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.620315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56118 (local address 127.12.222.194:36749)
0925 02:52:52.620430 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.620434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.620447 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:52:52.620618 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.620621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.620675 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.620774 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:52.621052 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.621775 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.622627 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.622956 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.623054 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.623264 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:52.623687 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:52.623792 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.623840 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.623885 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:52:52.633028 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.629856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:41339 (local address 127.12.222.193:43777)
0925 02:52:52.629970 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.629973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.630189 (+ 216us) server_negotiation.cc:407] Connection header received
0925 02:52:52.630330 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.630332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.630375 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.630477 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:52.630708 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.631152 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.631944 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.632158 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.632297 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.632608 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:52.632676 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:52.632754 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.632862 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.632906 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:52:52.633047 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.629760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:43777 (local address 127.12.222.194:41339)
0925 02:52:52.630106 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.630116 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.630230 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.630489 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.630491 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.630498 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:52:52.630614 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.630619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.631254 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.631258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.631810 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.631817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.632295 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.632299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.632386 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.632420 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:52:52.632759 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:52:52.632767 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.632850 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.632918 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":98,"threads_started":1}
I20260925 02:52:52.638849 15374 raft_consensus.cc:1261] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e [term 1 FOLLOWER]: Refusing update from remote peer 630879a0cbdc4d9ba586b0bc1439c387: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:52.638892 15091 raft_consensus.cc:1261] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 1 FOLLOWER]: Refusing update from remote peer 630879a0cbdc4d9ba586b0bc1439c387: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:52.639233 15449 consensus_queue.cc:1035] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:52.639377 15454 consensus_queue.cc:1035] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:52.661183 15488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.657738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:36749 (local address 127.12.222.193:51375)
0925 02:52:52.658039 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.658050 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.658156 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.658410 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.658412 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.658419 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:52:52.658559 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.658563 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.659369 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.659374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.659959 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.659967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.660454 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.660458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.660548 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.660587 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:52:52.660886 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:52:52.660894 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.660956 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.661030 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
I20260925 02:52:52.661163 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.657812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:51375 (local address 127.12.222.194:36749)
0925 02:52:52.657928 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.657933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.658112 (+ 179us) server_negotiation.cc:407] Connection header received
0925 02:52:52.658252 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.658255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.658299 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.658387 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:52.658653 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.659275 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.660071 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.660348 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.660444 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.660709 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:52.660780 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:52.660878 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.660968 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.661014 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:52:52.687672 14934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:52:52.687867 14934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:52:52.690444 14934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3: Sending DeleteTablet for 3 replicas of tablet a84af9354b204435b6c9d4caa264d1e4
I20260925 02:52:52.691099 15354 tablet_service.cc:1514] Processing DeleteTablet for tablet a84af9354b204435b6c9d4caa264d1e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:52 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:55092
I20260925 02:52:52.691339 15071 tablet_service.cc:1514] Processing DeleteTablet for tablet a84af9354b204435b6c9d4caa264d1e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:52 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:33178
I20260925 02:52:52.691699 15492 tablet_replica.cc:330] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e: stopping tablet replica
I20260925 02:52:52.691886 15213 tablet_service.cc:1514] Processing DeleteTablet for tablet a84af9354b204435b6c9d4caa264d1e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:52 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:56110
I20260925 02:52:52.692111 15492 raft_consensus.cc:2229] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:52:52.692373 15492 raft_consensus.cc:2258] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:52.693080 14934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:52:52.693465 14934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:52:52.694033 15492 ts_tablet_manager.cc:1905] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:52.694334 15494 tablet_replica.cc:330] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387: stopping tablet replica
I20260925 02:52:52.694584 15494 raft_consensus.cc:2229] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:52:52.694969 15494 raft_consensus.cc:2258] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:52.697122 15492 ts_tablet_manager.cc:1918] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:52:52.697193 15494 ts_tablet_manager.cc:1905] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:52.697218 15492 log.cc:1196] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/a84af9354b204435b6c9d4caa264d1e4
I20260925 02:52:52.697538 15492 ts_tablet_manager.cc:1939] T a84af9354b204435b6c9d4caa264d1e4 P 2a43a93680db4067a02fc6016c0e2a0e: Deleting consensus metadata
I20260925 02:52:52.698675 14918 catalog_manager.cc:4841] TS 2a43a93680db4067a02fc6016c0e2a0e (127.12.222.195:35437): tablet a84af9354b204435b6c9d4caa264d1e4 (table test-table [id=95c64a9954e34ba183e4a6ffb20eda25]) successfully deleted
I20260925 02:52:52.698742 15493 tablet_replica.cc:330] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab: stopping tablet replica
I20260925 02:52:52.698859 15213 tablet_service.cc:1467] Processing CreateTablet for tablet 9cff68dd11c243f38cdbc5d41e74c5cf (DEFAULT_TABLE table=test-table [id=e636f692fa6f46fd9ac434ad81868371]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:52.699011 15493 raft_consensus.cc:2229] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:52:52.699086 15213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cff68dd11c243f38cdbc5d41e74c5cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:52.699158 15493 raft_consensus.cc:2258] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:52.699116 15354 tablet_service.cc:1467] Processing CreateTablet for tablet 9cff68dd11c243f38cdbc5d41e74c5cf (DEFAULT_TABLE table=test-table [id=e636f692fa6f46fd9ac434ad81868371]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:52.699354 15354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cff68dd11c243f38cdbc5d41e74c5cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:52.700495 15071 tablet_service.cc:1467] Processing CreateTablet for tablet 9cff68dd11c243f38cdbc5d41e74c5cf (DEFAULT_TABLE table=test-table [id=e636f692fa6f46fd9ac434ad81868371]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:52.700734 15071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cff68dd11c243f38cdbc5d41e74c5cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:52.701145 15445 tablet_bootstrap.cc:492] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e: Bootstrap starting.
I20260925 02:52:52.701510 15443 tablet_bootstrap.cc:492] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387: Bootstrap starting.
I20260925 02:52:52.702059 15445 tablet_bootstrap.cc:654] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:52.702404 15443 tablet_bootstrap.cc:654] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:52.703272 15444 tablet_bootstrap.cc:492] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab: Bootstrap starting.
I20260925 02:52:52.704258 15444 tablet_bootstrap.cc:654] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:52.704494 15494 ts_tablet_manager.cc:1918] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:52:52.704581 15494 log.cc:1196] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/a84af9354b204435b6c9d4caa264d1e4
I20260925 02:52:52.704850 15494 ts_tablet_manager.cc:1939] T a84af9354b204435b6c9d4caa264d1e4 P 630879a0cbdc4d9ba586b0bc1439c387: Deleting consensus metadata
I20260925 02:52:52.705158 15493 ts_tablet_manager.cc:1905] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:52.705755 14920 catalog_manager.cc:4841] TS 630879a0cbdc4d9ba586b0bc1439c387 (127.12.222.194:36749): tablet a84af9354b204435b6c9d4caa264d1e4 (table test-table [id=95c64a9954e34ba183e4a6ffb20eda25]) successfully deleted
I20260925 02:52:52.707707 15493 ts_tablet_manager.cc:1918] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:52:52.707811 15493 log.cc:1196] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/wal/wals/a84af9354b204435b6c9d4caa264d1e4
I20260925 02:52:52.708103 15493 ts_tablet_manager.cc:1939] T a84af9354b204435b6c9d4caa264d1e4 P ab6d789eaaf140ec8119a3912c9e31ab: Deleting consensus metadata
I20260925 02:52:52.708900 15443 tablet_bootstrap.cc:492] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387: No bootstrap required, opened a new log
I20260925 02:52:52.709000 15443 ts_tablet_manager.cc:1397] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260925 02:52:52.709419 15443 raft_consensus.cc:348] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:52.709553 15443 raft_consensus.cc:374] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:52.709589 15443 raft_consensus.cc:729] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 630879a0cbdc4d9ba586b0bc1439c387, State: Initialized, Role: FOLLOWER
I20260925 02:52:52.709662 14919 catalog_manager.cc:4841] TS ab6d789eaaf140ec8119a3912c9e31ab (127.12.222.193:43777): tablet a84af9354b204435b6c9d4caa264d1e4 (table test-table [id=95c64a9954e34ba183e4a6ffb20eda25]) successfully deleted
I20260925 02:52:52.709697 15443 consensus_queue.cc:260] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:52.709942 15443 ts_tablet_manager.cc:1428] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:52:52.710472 15444 tablet_bootstrap.cc:492] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab: No bootstrap required, opened a new log
I20260925 02:52:52.710552 15444 ts_tablet_manager.cc:1397] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260925 02:52:52.710953 15444 raft_consensus.cc:348] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:52.711051 15444 raft_consensus.cc:374] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:52.711081 15444 raft_consensus.cc:729] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6d789eaaf140ec8119a3912c9e31ab, State: Initialized, Role: FOLLOWER
I20260925 02:52:52.711176 15444 consensus_queue.cc:260] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:52.711392 15444 ts_tablet_manager.cc:1428] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:52:52.713070 15445 tablet_bootstrap.cc:492] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e: No bootstrap required, opened a new log
I20260925 02:52:52.713152 15445 ts_tablet_manager.cc:1397] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20260925 02:52:52.713557 15445 raft_consensus.cc:348] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:52.713654 15445 raft_consensus.cc:374] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:52.713683 15445 raft_consensus.cc:729] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a43a93680db4067a02fc6016c0e2a0e, State: Initialized, Role: FOLLOWER
I20260925 02:52:52.713793 15445 consensus_queue.cc:260] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:52.714028 15445 ts_tablet_manager.cc:1428] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:52:52.768441 15451 raft_consensus.cc:482] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:52:52.768546 15451 raft_consensus.cc:504] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:52.769559 15451 leader_election.cc:288] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab6d789eaaf140ec8119a3912c9e31ab (127.12.222.193:43777), 630879a0cbdc4d9ba586b0bc1439c387 (127.12.222.194:36749)
I20260925 02:52:52.773999 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.769859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:51783 (local address 127.12.222.193:43777)
0925 02:52:52.770065 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.770074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.770119 (+ 45us) server_negotiation.cc:407] Connection header received
0925 02:52:52.770329 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.770332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.770393 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.770454 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:52.770915 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.771714 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.772815 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.773424 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.773543 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.773596 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.773815 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.773888 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260925 02:52:52.774019 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.769743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:43777 (local address 127.12.222.195:51783)
0925 02:52:52.770036 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.770051 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.770199 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.770609 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.770610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.770654 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:52.770797 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.770806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.771836 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.771839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.772688 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.772698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.773549 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.773556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.773676 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.773706 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.773793 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.773863 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260925 02:52:52.774536 15091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cff68dd11c243f38cdbc5d41e74c5cf" candidate_uuid: "2a43a93680db4067a02fc6016c0e2a0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" is_pre_election: true
I20260925 02:52:52.774690 15091 raft_consensus.cc:2454] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a43a93680db4067a02fc6016c0e2a0e in term 0.
I20260925 02:52:52.774906 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.770251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:37743 (local address 127.12.222.194:36749)
0925 02:52:52.770370 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.770374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.770390 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:52:52.770525 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.770528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.770576 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.770657 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:52.771869 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.772646 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.773659 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.774242 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.774354 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.774399 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.774710 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.774779 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:52:52.775059 15305 leader_election.cc:302] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a43a93680db4067a02fc6016c0e2a0e, ab6d789eaaf140ec8119a3912c9e31ab; no voters:
I20260925 02:52:52.775295 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.769891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:36749 (local address 127.12.222.195:37743)
0925 02:52:52.770333 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.770346 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.770414 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.770686 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.770689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.770701 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:52.770808 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.770814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.772764 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.772769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.773509 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.773520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.774462 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.774466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.774571 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.774593 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.775094 (+ 501us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.775167 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":22,"thread_start_us":79,"threads_started":1}
I20260925 02:52:52.775338 15451 raft_consensus.cc:2781] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:52:52.775405 15451 raft_consensus.cc:482] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:52:52.775451 15451 raft_consensus.cc:3037] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:52.775717 15233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cff68dd11c243f38cdbc5d41e74c5cf" candidate_uuid: "2a43a93680db4067a02fc6016c0e2a0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "630879a0cbdc4d9ba586b0bc1439c387" is_pre_election: true
I20260925 02:52:52.775846 15233 raft_consensus.cc:2454] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a43a93680db4067a02fc6016c0e2a0e in term 0.
I20260925 02:52:52.776698 15451 raft_consensus.cc:504] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:52.777081 15451 leader_election.cc:288] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [CANDIDATE]: Term 1 election: Requested vote from peers ab6d789eaaf140ec8119a3912c9e31ab (127.12.222.193:43777), 630879a0cbdc4d9ba586b0bc1439c387 (127.12.222.194:36749)
I20260925 02:52:52.777356 15091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cff68dd11c243f38cdbc5d41e74c5cf" candidate_uuid: "2a43a93680db4067a02fc6016c0e2a0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6d789eaaf140ec8119a3912c9e31ab"
I20260925 02:52:52.777398 15233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cff68dd11c243f38cdbc5d41e74c5cf" candidate_uuid: "2a43a93680db4067a02fc6016c0e2a0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "630879a0cbdc4d9ba586b0bc1439c387"
I20260925 02:52:52.777479 15091 raft_consensus.cc:3037] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:52.777520 15233 raft_consensus.cc:3037] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:52.778467 15233 raft_consensus.cc:2454] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a43a93680db4067a02fc6016c0e2a0e in term 1.
I20260925 02:52:52.778555 15091 raft_consensus.cc:2454] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a43a93680db4067a02fc6016c0e2a0e in term 1.
I20260925 02:52:52.778743 15306 leader_election.cc:302] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a43a93680db4067a02fc6016c0e2a0e, 630879a0cbdc4d9ba586b0bc1439c387; no voters:
I20260925 02:52:52.778959 15451 raft_consensus.cc:2781] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:52.779141 15451 raft_consensus.cc:686] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [term 1 LEADER]: Becoming Leader. State: Replica: 2a43a93680db4067a02fc6016c0e2a0e, State: Running, Role: LEADER
I20260925 02:52:52.779368 15451 consensus_queue.cc:237] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:52.781608 14934 catalog_manager.cc:5495] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a43a93680db4067a02fc6016c0e2a0e (127.12.222.195). New cstate: current_term: 1 leader_uuid: "2a43a93680db4067a02fc6016c0e2a0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } health_report { overall_health: HEALTHY } } }
I20260925 02:52:52.812961 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.809090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55106 (local address 127.12.222.195:35437)
0925 02:52:52.809200 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.809205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.809218 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:52:52.809328 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.809331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.809391 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.809511 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:52.809804 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.810264 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.811397 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.811605 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.811670 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.812005 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:52.812469 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:52.812571 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.812714 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.812761 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:52:52.823230 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.819748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:53735 (local address 127.12.222.193:43777)
0925 02:52:52.819872 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.819877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.820065 (+ 188us) server_negotiation.cc:407] Connection header received
0925 02:52:52.820197 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.820200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.820293 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.820365 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:52.820615 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.821166 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.822076 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.822369 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.822510 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.822835 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:52.822893 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:52.822948 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.823094 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.823136 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:52:52.823319 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.819665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:43777 (local address 127.12.222.195:53735)
0925 02:52:52.819982 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.819993 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.820116 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.820387 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.820390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.820397 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:52:52.820503 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.820516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.821283 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.821286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.821957 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.821965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.822544 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.822548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.822647 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.822698 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:52:52.822991 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:52:52.822999 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.823076 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.823157 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":93,"threads_started":1}
I20260925 02:52:52.828691 15091 raft_consensus.cc:1261] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab [term 1 FOLLOWER]: Refusing update from remote peer 2a43a93680db4067a02fc6016c0e2a0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:52.828799 15233 raft_consensus.cc:1261] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 FOLLOWER]: Refusing update from remote peer 2a43a93680db4067a02fc6016c0e2a0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:52.829116 15500 consensus_queue.cc:1035] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:52.829294 15451 consensus_queue.cc:1035] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:52.848412 15488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.844743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:35437 (local address 127.12.222.193:53309)
0925 02:52:52.844854 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:52.844867 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:52.844942 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:52.845335 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:52.845338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:52.845346 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:52:52.845432 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.845437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.846365 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.846367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.847025 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:52.847034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.847637 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.847642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:52.847733 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.847764 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:52:52.848089 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:52:52.848098 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 02:52:52.848167 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 02:52:52.848283 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 02:52:52.848477 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:52.845042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:53309 (local address 127.12.222.195:35437)
0925 02:52:52.845146 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:52.845150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:52.845165 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:52:52.845204 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:52.845207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:52.845257 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:52.845326 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:52.845548 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.846255 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.847140 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:52.847471 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:52.847564 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:52.847868 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:52.847955 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:52.848033 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:52.848298 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 02:52:52.848351 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260925 02:52:52.849107 15474 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 02:52:54.406827 14934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:52:54.406998 14934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:52:54.409183 14934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3: Sending DeleteTablet for 3 replicas of tablet 9cff68dd11c243f38cdbc5d41e74c5cf
I20260925 02:52:54.409647 15071 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cff68dd11c243f38cdbc5d41e74c5cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:54 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:33178
I20260925 02:52:54.409838 15213 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cff68dd11c243f38cdbc5d41e74c5cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:54 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:56110
I20260925 02:52:54.409984 15530 tablet_replica.cc:330] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab: stopping tablet replica
I20260925 02:52:54.410126 15530 raft_consensus.cc:2229] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:52:54.410151 15354 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cff68dd11c243f38cdbc5d41e74c5cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:54 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:55092
I20260925 02:52:54.410260 15530 raft_consensus.cc:2258] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:54.410501 15531 tablet_replica.cc:330] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387: stopping tablet replica
I20260925 02:52:54.410620 15531 raft_consensus.cc:2229] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:52:54.410719 15531 raft_consensus.cc:2258] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:54.410915 15530 ts_tablet_manager.cc:1905] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:54.410908 15532 tablet_replica.cc:330] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e: stopping tablet replica
I20260925 02:52:54.411023 15532 raft_consensus.cc:2229] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:52:54.411309 15532 raft_consensus.cc:2258] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:54.411912 15531 ts_tablet_manager.cc:1905] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:54.413939 15530 ts_tablet_manager.cc:1918] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:52:54.414017 15530 log.cc:1196] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/wal/wals/9cff68dd11c243f38cdbc5d41e74c5cf
I20260925 02:52:54.414275 15530 ts_tablet_manager.cc:1939] T 9cff68dd11c243f38cdbc5d41e74c5cf P ab6d789eaaf140ec8119a3912c9e31ab: Deleting consensus metadata
I20260925 02:52:54.414682 15531 ts_tablet_manager.cc:1918] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:52:54.414760 15531 log.cc:1196] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/9cff68dd11c243f38cdbc5d41e74c5cf
I20260925 02:52:54.414959 14919 catalog_manager.cc:4841] TS ab6d789eaaf140ec8119a3912c9e31ab (127.12.222.193:43777): tablet 9cff68dd11c243f38cdbc5d41e74c5cf (table test-table [id=e636f692fa6f46fd9ac434ad81868371]) successfully deleted
I20260925 02:52:54.415056 15531 ts_tablet_manager.cc:1939] T 9cff68dd11c243f38cdbc5d41e74c5cf P 630879a0cbdc4d9ba586b0bc1439c387: Deleting consensus metadata
I20260925 02:52:54.415700 14920 catalog_manager.cc:4841] TS 630879a0cbdc4d9ba586b0bc1439c387 (127.12.222.194:36749): tablet 9cff68dd11c243f38cdbc5d41e74c5cf (table test-table [id=e636f692fa6f46fd9ac434ad81868371]) successfully deleted
I20260925 02:52:54.416018 15532 ts_tablet_manager.cc:1905] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:54.418465 15532 ts_tablet_manager.cc:1918] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:52:54.418529 15532 log.cc:1196] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/9cff68dd11c243f38cdbc5d41e74c5cf
I20260925 02:52:54.418792 15532 ts_tablet_manager.cc:1939] T 9cff68dd11c243f38cdbc5d41e74c5cf P 2a43a93680db4067a02fc6016c0e2a0e: Deleting consensus metadata
I20260925 02:52:54.419524 14918 catalog_manager.cc:4841] TS 2a43a93680db4067a02fc6016c0e2a0e (127.12.222.195:35437): tablet 9cff68dd11c243f38cdbc5d41e74c5cf (table test-table [id=e636f692fa6f46fd9ac434ad81868371]) successfully deleted
Sep 25 02:52:54 dist-test-slave-252r krb5kdc[14888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:52:54 dist-test-slave-252r krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:54.443059 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:54.429079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37224 (local address 127.12.222.254:38297)
0925 02:52:54.429333 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:54.429341 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:54.429550 (+ 209us) server_negotiation.cc:407] Connection header received
0925 02:52:54.429639 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:54.429641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:54.429710 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:54.429796 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:54.430474 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:54.431014 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:54.431726 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:54.431920 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:54.431987 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:54.440442 (+ 8455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:54.440450 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:54.440452 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:54.440474 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:54.441669 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:54.442007 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:54.442010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:54.442011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:54.442078 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:54.442284 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:54.442286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:54.442287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:54.442439 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:54.442539 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:54.442744 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 02:52:54.442837 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
I20260925 02:52:54.445322 14934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:52:54.445678 14934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:52:54.450053 15213 tablet_service.cc:1467] Processing CreateTablet for tablet afb3d796717145ef9ffd1a8742e20bc7 (DEFAULT_TABLE table=test-table [id=37fb44bccfdb439ca92e5c10a0151718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:54.450110 15354 tablet_service.cc:1467] Processing CreateTablet for tablet afb3d796717145ef9ffd1a8742e20bc7 (DEFAULT_TABLE table=test-table [id=37fb44bccfdb439ca92e5c10a0151718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:54.450181 15071 tablet_service.cc:1467] Processing CreateTablet for tablet afb3d796717145ef9ffd1a8742e20bc7 (DEFAULT_TABLE table=test-table [id=37fb44bccfdb439ca92e5c10a0151718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:54.450354 15354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb3d796717145ef9ffd1a8742e20bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:54.450354 15213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb3d796717145ef9ffd1a8742e20bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:54.450470 15071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb3d796717145ef9ffd1a8742e20bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:54.452332 15541 tablet_bootstrap.cc:492] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab: Bootstrap starting.
I20260925 02:52:54.452700 15543 tablet_bootstrap.cc:492] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387: Bootstrap starting.
I20260925 02:52:54.453048 15542 tablet_bootstrap.cc:492] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e: Bootstrap starting.
I20260925 02:52:54.453289 15541 tablet_bootstrap.cc:654] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:54.453589 15543 tablet_bootstrap.cc:654] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:54.453882 15542 tablet_bootstrap.cc:654] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:54.454541 15541 tablet_bootstrap.cc:492] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab: No bootstrap required, opened a new log
I20260925 02:52:54.454636 15541 ts_tablet_manager.cc:1397] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:54.454710 15543 tablet_bootstrap.cc:492] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387: No bootstrap required, opened a new log
I20260925 02:52:54.454795 15543 ts_tablet_manager.cc:1397] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:52:54.455054 15541 raft_consensus.cc:348] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:54.455161 15541 raft_consensus.cc:374] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:54.455188 15542 tablet_bootstrap.cc:492] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e: No bootstrap required, opened a new log
I20260925 02:52:54.455193 15541 raft_consensus.cc:729] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6d789eaaf140ec8119a3912c9e31ab, State: Initialized, Role: FOLLOWER
I20260925 02:52:54.455256 15542 ts_tablet_manager.cc:1397] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:54.455307 15541 consensus_queue.cc:260] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:54.455360 15543 raft_consensus.cc:348] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:54.455477 15543 raft_consensus.cc:374] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:54.455511 15543 raft_consensus.cc:729] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 630879a0cbdc4d9ba586b0bc1439c387, State: Initialized, Role: FOLLOWER
I20260925 02:52:54.455562 15541 ts_tablet_manager.cc:1428] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:52:54.455605 15542 raft_consensus.cc:348] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:54.455705 15542 raft_consensus.cc:374] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:54.455660 15543 consensus_queue.cc:260] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:54.455739 15542 raft_consensus.cc:729] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a43a93680db4067a02fc6016c0e2a0e, State: Initialized, Role: FOLLOWER
I20260925 02:52:54.455858 15542 consensus_queue.cc:260] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:54.456207 15542 ts_tablet_manager.cc:1428] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:52:54.456503 15543 ts_tablet_manager.cc:1428] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:54.547820 15547 raft_consensus.cc:482] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:52:54.547960 15547 raft_consensus.cc:504] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:54.548450 15547 leader_election.cc:288] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab6d789eaaf140ec8119a3912c9e31ab (127.12.222.193:43777), 2a43a93680db4067a02fc6016c0e2a0e (127.12.222.195:35437)
I20260925 02:52:54.548957 15091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb3d796717145ef9ffd1a8742e20bc7" candidate_uuid: "630879a0cbdc4d9ba586b0bc1439c387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" is_pre_election: true
I20260925 02:52:54.548977 15373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb3d796717145ef9ffd1a8742e20bc7" candidate_uuid: "630879a0cbdc4d9ba586b0bc1439c387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a43a93680db4067a02fc6016c0e2a0e" is_pre_election: true
I20260925 02:52:54.549111 15373 raft_consensus.cc:2454] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 630879a0cbdc4d9ba586b0bc1439c387 in term 0.
I20260925 02:52:54.549111 15091 raft_consensus.cc:2454] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 630879a0cbdc4d9ba586b0bc1439c387 in term 0.
I20260925 02:52:54.549459 15164 leader_election.cc:302] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 630879a0cbdc4d9ba586b0bc1439c387, ab6d789eaaf140ec8119a3912c9e31ab; no voters:
I20260925 02:52:54.549621 15547 raft_consensus.cc:2781] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:52:54.549680 15547 raft_consensus.cc:482] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:52:54.549715 15547 raft_consensus.cc:3037] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:54.550791 15547 raft_consensus.cc:504] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:54.551184 15547 leader_election.cc:288] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [CANDIDATE]: Term 1 election: Requested vote from peers ab6d789eaaf140ec8119a3912c9e31ab (127.12.222.193:43777), 2a43a93680db4067a02fc6016c0e2a0e (127.12.222.195:35437)
I20260925 02:52:54.551468 15091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb3d796717145ef9ffd1a8742e20bc7" candidate_uuid: "630879a0cbdc4d9ba586b0bc1439c387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6d789eaaf140ec8119a3912c9e31ab"
I20260925 02:52:54.551621 15091 raft_consensus.cc:3037] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:54.551570 15373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb3d796717145ef9ffd1a8742e20bc7" candidate_uuid: "630879a0cbdc4d9ba586b0bc1439c387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a43a93680db4067a02fc6016c0e2a0e"
I20260925 02:52:54.551715 15373 raft_consensus.cc:3037] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:54.552686 15091 raft_consensus.cc:2454] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 630879a0cbdc4d9ba586b0bc1439c387 in term 1.
I20260925 02:52:54.552958 15164 leader_election.cc:302] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 630879a0cbdc4d9ba586b0bc1439c387, ab6d789eaaf140ec8119a3912c9e31ab; no voters:
I20260925 02:52:54.553143 15547 raft_consensus.cc:2781] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:54.553166 15373 raft_consensus.cc:2454] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 630879a0cbdc4d9ba586b0bc1439c387 in term 1.
I20260925 02:52:54.553421 15547 raft_consensus.cc:686] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 LEADER]: Becoming Leader. State: Replica: 630879a0cbdc4d9ba586b0bc1439c387, State: Running, Role: LEADER
I20260925 02:52:54.553566 15547 consensus_queue.cc:237] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:54.555444 14933 catalog_manager.cc:5495] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 reported cstate change: term changed from 0 to 1, leader changed from <none> to 630879a0cbdc4d9ba586b0bc1439c387 (127.12.222.194). New cstate: current_term: 1 leader_uuid: "630879a0cbdc4d9ba586b0bc1439c387" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } health_report { overall_health: UNKNOWN } } }
I20260925 02:52:54.582909 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:54.579861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56132 (local address 127.12.222.194:36749)
0925 02:52:54.580072 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:54.580078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:54.580088 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:52:54.580120 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:54.580122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:54.580156 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:54.580261 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:54.580502 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:54.580987 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:54.581827 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:54.582162 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:54.582266 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:54.582481 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:54.582554 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:54.582636 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:54.582729 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 02:52:54.582779 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
I20260925 02:52:54.587920 15091 raft_consensus.cc:1261] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab [term 1 FOLLOWER]: Refusing update from remote peer 630879a0cbdc4d9ba586b0bc1439c387: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:54.588554 15547 consensus_queue.cc:1035] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:54.588604 15373 raft_consensus.cc:1261] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e [term 1 FOLLOWER]: Refusing update from remote peer 630879a0cbdc4d9ba586b0bc1439c387: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:54.588980 15547 consensus_queue.cc:1035] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:54.615646 14932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:52:54.615775 14932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:52:54.617563 14932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3: Sending DeleteTablet for 3 replicas of tablet afb3d796717145ef9ffd1a8742e20bc7
I20260925 02:52:54.618064 15071 tablet_service.cc:1514] Processing DeleteTablet for tablet afb3d796717145ef9ffd1a8742e20bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:54 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:33178
I20260925 02:52:54.618168 15354 tablet_service.cc:1514] Processing DeleteTablet for tablet afb3d796717145ef9ffd1a8742e20bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:54 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:55092
I20260925 02:52:54.618185 15213 tablet_service.cc:1514] Processing DeleteTablet for tablet afb3d796717145ef9ffd1a8742e20bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:54 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:56110
I20260925 02:52:54.618259 15530 tablet_replica.cc:330] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab: stopping tablet replica
I20260925 02:52:54.618363 15532 tablet_replica.cc:330] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e: stopping tablet replica
I20260925 02:52:54.618366 15530 raft_consensus.cc:2229] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:52:54.618486 15530 raft_consensus.cc:2258] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:54.618491 15532 raft_consensus.cc:2229] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:52:54.618611 15532 raft_consensus.cc:2258] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:54.618779 15531 tablet_replica.cc:330] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387: stopping tablet replica
I20260925 02:52:54.618911 15531 raft_consensus.cc:2229] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:52:54.619100 15531 raft_consensus.cc:2258] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:54.619174 15530 ts_tablet_manager.cc:1905] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:54.619700 15531 ts_tablet_manager.cc:1905] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:54.619709 15532 ts_tablet_manager.cc:1905] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:54.621922 15530 ts_tablet_manager.cc:1918] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:52:54.621937 15532 ts_tablet_manager.cc:1918] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:52:54.622012 15530 log.cc:1196] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-0/wal/wals/afb3d796717145ef9ffd1a8742e20bc7
I20260925 02:52:54.622026 15532 log.cc:1196] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/afb3d796717145ef9ffd1a8742e20bc7
I20260925 02:52:54.622290 15530 ts_tablet_manager.cc:1939] T afb3d796717145ef9ffd1a8742e20bc7 P ab6d789eaaf140ec8119a3912c9e31ab: Deleting consensus metadata
I20260925 02:52:54.622304 15532 ts_tablet_manager.cc:1939] T afb3d796717145ef9ffd1a8742e20bc7 P 2a43a93680db4067a02fc6016c0e2a0e: Deleting consensus metadata
I20260925 02:52:54.622323 15531 ts_tablet_manager.cc:1918] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:52:54.622393 15531 log.cc:1196] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/afb3d796717145ef9ffd1a8742e20bc7
I20260925 02:52:54.622669 15531 ts_tablet_manager.cc:1939] T afb3d796717145ef9ffd1a8742e20bc7 P 630879a0cbdc4d9ba586b0bc1439c387: Deleting consensus metadata
I20260925 02:52:54.622912 14918 catalog_manager.cc:4841] TS 2a43a93680db4067a02fc6016c0e2a0e (127.12.222.195:35437): tablet afb3d796717145ef9ffd1a8742e20bc7 (table test-table [id=37fb44bccfdb439ca92e5c10a0151718]) successfully deleted
I20260925 02:52:54.622991 14919 catalog_manager.cc:4841] TS ab6d789eaaf140ec8119a3912c9e31ab (127.12.222.193:43777): tablet afb3d796717145ef9ffd1a8742e20bc7 (table test-table [id=37fb44bccfdb439ca92e5c10a0151718]) successfully deleted
I20260925 02:52:54.623255 14920 catalog_manager.cc:4841] TS 630879a0cbdc4d9ba586b0bc1439c387 (127.12.222.194:36749): tablet afb3d796717145ef9ffd1a8742e20bc7 (table test-table [id=37fb44bccfdb439ca92e5c10a0151718]) successfully deleted
I20260925 02:52:54.623257 14932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:52:54.623652 14932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:52:54.627288 15071 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa278ba5a244316bbbcdf2b476025e4 (DEFAULT_TABLE table=test-table [id=963f29f16d104867a57bb20c96fd5518]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:54.627372 15213 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa278ba5a244316bbbcdf2b476025e4 (DEFAULT_TABLE table=test-table [id=963f29f16d104867a57bb20c96fd5518]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:54.627535 15071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa278ba5a244316bbbcdf2b476025e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:54.627557 15213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa278ba5a244316bbbcdf2b476025e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:54.627527 15354 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa278ba5a244316bbbcdf2b476025e4 (DEFAULT_TABLE table=test-table [id=963f29f16d104867a57bb20c96fd5518]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:54.627763 15354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa278ba5a244316bbbcdf2b476025e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:54.629679 15541 tablet_bootstrap.cc:492] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab: Bootstrap starting.
I20260925 02:52:54.629804 15542 tablet_bootstrap.cc:492] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e: Bootstrap starting.
I20260925 02:52:54.630422 15542 tablet_bootstrap.cc:654] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:54.630487 15541 tablet_bootstrap.cc:654] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:54.631300 15542 tablet_bootstrap.cc:492] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e: No bootstrap required, opened a new log
I20260925 02:52:54.631404 15542 ts_tablet_manager.cc:1397] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 02:52:54.631580 15541 tablet_bootstrap.cc:492] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab: No bootstrap required, opened a new log
I20260925 02:52:54.631645 15541 ts_tablet_manager.cc:1397] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:52:54.631790 15542 raft_consensus.cc:348] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:54.631891 15542 raft_consensus.cc:374] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:54.631929 15542 raft_consensus.cc:729] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a43a93680db4067a02fc6016c0e2a0e, State: Initialized, Role: FOLLOWER
I20260925 02:52:54.632005 15541 raft_consensus.cc:348] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:54.632054 15542 consensus_queue.cc:260] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:54.632105 15541 raft_consensus.cc:374] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:54.632156 15541 raft_consensus.cc:729] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6d789eaaf140ec8119a3912c9e31ab, State: Initialized, Role: FOLLOWER
I20260925 02:52:54.632320 15542 ts_tablet_manager.cc:1428] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:52:54.632292 15541 consensus_queue.cc:260] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:54.632453 15543 tablet_bootstrap.cc:492] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387: Bootstrap starting.
I20260925 02:52:54.632543 15541 ts_tablet_manager.cc:1428] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:52:54.633234 15543 tablet_bootstrap.cc:654] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:54.634186 15543 tablet_bootstrap.cc:492] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387: No bootstrap required, opened a new log
I20260925 02:52:54.634268 15543 ts_tablet_manager.cc:1397] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:52:54.634567 15543 raft_consensus.cc:348] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:54.634670 15543 raft_consensus.cc:374] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:54.634709 15543 raft_consensus.cc:729] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 630879a0cbdc4d9ba586b0bc1439c387, State: Initialized, Role: FOLLOWER
I20260925 02:52:54.634816 15543 consensus_queue.cc:260] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:54.635078 15543 ts_tablet_manager.cc:1428] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:52:54.677627 15517 raft_consensus.cc:482] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:52:54.677742 15517 raft_consensus.cc:504] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:54.678228 15517 leader_election.cc:288] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 630879a0cbdc4d9ba586b0bc1439c387 (127.12.222.194:36749), 2a43a93680db4067a02fc6016c0e2a0e (127.12.222.195:35437)
I20260925 02:52:54.678666 15233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa278ba5a244316bbbcdf2b476025e4" candidate_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "630879a0cbdc4d9ba586b0bc1439c387" is_pre_election: true
I20260925 02:52:54.678675 15373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa278ba5a244316bbbcdf2b476025e4" candidate_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a43a93680db4067a02fc6016c0e2a0e" is_pre_election: true
I20260925 02:52:54.678834 15233 raft_consensus.cc:2454] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab6d789eaaf140ec8119a3912c9e31ab in term 0.
I20260925 02:52:54.678858 15373 raft_consensus.cc:2454] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab6d789eaaf140ec8119a3912c9e31ab in term 0.
I20260925 02:52:54.679169 15022 leader_election.cc:302] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a43a93680db4067a02fc6016c0e2a0e, ab6d789eaaf140ec8119a3912c9e31ab; no voters:
I20260925 02:52:54.679353 15517 raft_consensus.cc:2781] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:52:54.679409 15517 raft_consensus.cc:482] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:52:54.679461 15517 raft_consensus.cc:3037] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:54.680455 15517 raft_consensus.cc:504] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:54.680725 15517 leader_election.cc:288] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [CANDIDATE]: Term 1 election: Requested vote from peers 630879a0cbdc4d9ba586b0bc1439c387 (127.12.222.194:36749), 2a43a93680db4067a02fc6016c0e2a0e (127.12.222.195:35437)
I20260925 02:52:54.681044 15233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa278ba5a244316bbbcdf2b476025e4" candidate_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "630879a0cbdc4d9ba586b0bc1439c387"
I20260925 02:52:54.681084 15373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa278ba5a244316bbbcdf2b476025e4" candidate_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a43a93680db4067a02fc6016c0e2a0e"
I20260925 02:52:54.681161 15233 raft_consensus.cc:3037] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:54.681185 15373 raft_consensus.cc:3037] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:54.682253 15373 raft_consensus.cc:2454] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6d789eaaf140ec8119a3912c9e31ab in term 1.
I20260925 02:52:54.682253 15233 raft_consensus.cc:2454] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6d789eaaf140ec8119a3912c9e31ab in term 1.
I20260925 02:52:54.682507 15024 leader_election.cc:302] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 630879a0cbdc4d9ba586b0bc1439c387, ab6d789eaaf140ec8119a3912c9e31ab; no voters:
I20260925 02:52:54.682708 15517 raft_consensus.cc:2781] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:54.682814 15517 raft_consensus.cc:686] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 1 LEADER]: Becoming Leader. State: Replica: ab6d789eaaf140ec8119a3912c9e31ab, State: Running, Role: LEADER
I20260925 02:52:54.682947 15517 consensus_queue.cc:237] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } }
I20260925 02:52:54.684441 14933 catalog_manager.cc:5495] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab reported cstate change: term changed from 0 to 1, leader changed from <none> to ab6d789eaaf140ec8119a3912c9e31ab (127.12.222.193). New cstate: current_term: 1 leader_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6d789eaaf140ec8119a3912c9e31ab" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 43777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 } health_report { overall_health: UNKNOWN } } }
I20260925 02:52:54.702067 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:54.698913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.12.222.193:43777)
0925 02:52:54.699155 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:54.699168 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:54.699183 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:52:54.699224 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:54.699227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:54.699274 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:54.699359 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:54.699591 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:54.700109 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:54.700999 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:54.701272 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:54.701379 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:54.701622 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:54.701698 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:54.701776 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:54.701867 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 02:52:54.701918 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
I20260925 02:52:54.709860 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:54.707008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:53235 (local address 127.12.222.193:43777)
0925 02:52:54.707118 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:54.707123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:54.707212 (+ 89us) server_negotiation.cc:407] Connection header received
0925 02:52:54.707338 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:54.707340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:54.707379 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:54.707460 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:54.707665 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:54.708087 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:54.708937 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:54.709140 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:54.709253 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:54.709498 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:54.709556 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:54.709624 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:54.709706 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 02:52:54.709742 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:52:54.709878 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:54.706851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:43777 (local address 127.12.222.193:53235)
0925 02:52:54.707118 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:54.707132 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:54.707244 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:54.707469 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:54.707471 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:54.707476 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:52:54.707555 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:54.707560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:54.708210 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:54.708216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:54.708821 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:54.708829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:54.709252 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:54.709255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:54.709347 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:54.709389 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:52:54.709637 (+ 248us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:52:54.709644 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:52:54.709698 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 02:52:54.709767 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":2,"thread_start_us":116,"threads_started":1}
I20260925 02:52:54.714473 15373 raft_consensus.cc:1261] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e [term 1 FOLLOWER]: Refusing update from remote peer ab6d789eaaf140ec8119a3912c9e31ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:54.714505 15233 raft_consensus.cc:1261] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 FOLLOWER]: Refusing update from remote peer ab6d789eaaf140ec8119a3912c9e31ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:54.714857 15509 consensus_queue.cc:1035] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a43a93680db4067a02fc6016c0e2a0e" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:54.715018 15512 consensus_queue.cc:1035] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "630879a0cbdc4d9ba586b0bc1439c387" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:52:54.727138 15510 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 02:52:56.275454 14932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:52:56.275591 14932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:52:56.277675 14932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee398adc75504eb69e9898e86446a9a3: Sending DeleteTablet for 3 replicas of tablet 1fa278ba5a244316bbbcdf2b476025e4
I20260925 02:52:56.278162 15071 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa278ba5a244316bbbcdf2b476025e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:56 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:33178
I20260925 02:52:56.278204 15213 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa278ba5a244316bbbcdf2b476025e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:56 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:56110
I20260925 02:52:56.278389 15354 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa278ba5a244316bbbcdf2b476025e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:56 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:55092
I20260925 02:52:56.278472 15586 tablet_replica.cc:330] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab: stopping tablet replica
I20260925 02:52:56.278622 15586 raft_consensus.cc:2229] T 1fa278ba5a244316bbbcdf2b476025e4 P ab6d789eaaf140ec8119a3912c9e31ab [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:52:56.278729 15588 tablet_replica.cc:330] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e: stopping tablet replica
I20260925 02:52:56.278757 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 15006
I20260925 02:52:56.279665 15587 tablet_replica.cc:330] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387: stopping tablet replica
I20260925 02:52:56.279783 15587 raft_consensus.cc:2229] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:52:56.279876 15587 raft_consensus.cc:2258] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:56.280768 15587 ts_tablet_manager.cc:1905] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:56.281355 15588 raft_consensus.cc:2229] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:52:56.281490 15588 raft_consensus.cc:2258] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:56.282052 15588 ts_tablet_manager.cc:1905] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:56.283488 15587 ts_tablet_manager.cc:1918] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:52:56.283629 15587 log.cc:1196] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/1fa278ba5a244316bbbcdf2b476025e4
I20260925 02:52:56.283964 15587 ts_tablet_manager.cc:1939] T 1fa278ba5a244316bbbcdf2b476025e4 P 630879a0cbdc4d9ba586b0bc1439c387: Deleting consensus metadata
I20260925 02:52:56.284720 15588 ts_tablet_manager.cc:1918] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:52:56.284690 14920 catalog_manager.cc:4841] TS 630879a0cbdc4d9ba586b0bc1439c387 (127.12.222.194:36749): tablet 1fa278ba5a244316bbbcdf2b476025e4 (table test-table [id=963f29f16d104867a57bb20c96fd5518]) successfully deleted
I20260925 02:52:56.284813 15588 log.cc:1196] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/1fa278ba5a244316bbbcdf2b476025e4
I20260925 02:52:56.285084 15588 ts_tablet_manager.cc:1939] T 1fa278ba5a244316bbbcdf2b476025e4 P 2a43a93680db4067a02fc6016c0e2a0e: Deleting consensus metadata
I20260925 02:52:56.285691 14918 catalog_manager.cc:4841] TS 2a43a93680db4067a02fc6016c0e2a0e (127.12.222.195:35437): tablet 1fa278ba5a244316bbbcdf2b476025e4 (table test-table [id=963f29f16d104867a57bb20c96fd5518]) successfully deleted
W20260925 02:52:56.287170 14919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.222.193:43777 (error 108)
I20260925 02:52:56.287248 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 15147
I20260925 02:52:56.288851 15590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:56.288454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:43777 (local address 127.0.0.1:33208)
0925 02:52:56.288683 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:56.288763 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.193:43777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
W20260925 02:52:56.289023 14919 catalog_manager.cc:4584] TS ab6d789eaaf140ec8119a3912c9e31ab (127.12.222.193:43777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1fa278ba5a244316bbbcdf2b476025e4: Network error: Client connection negotiation failed: client connection to 127.12.222.193:43777: connect: Connection refused (error 111)
I20260925 02:52:56.295470 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 15288
I20260925 02:52:56.302189 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 14901
2026-09-25T02:52:56Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5959 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:56 dist-test-slave-252r krb5kdc[15593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:56 dist-test-slave-252r krb5kdc[15593](info): set up 2 sockets
Sep 25 02:52:56 dist-test-slave-252r krb5kdc[15593](info): commencing operation
krb5kdc: starting...
W20260925 02:52:57.284037 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:52:57 dist-test-slave-252r krb5kdc[15593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:52:57.391664 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:46199
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:33515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:46199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:57.492111 15606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:57.492409 15606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:57.492483 15606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:57.496160 15606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:57.496205 15606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:57.496265 15606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:57.496301 15606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:57.496326 15606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:57.501260 15606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:33515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:46199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:46199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:57.502121 15606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:57.503029 15606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:57.510618 15612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:57.510716 15606 server_base.cc:1034] running on GCE node
W20260925 02:52:57.510679 15611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:57.510634 15614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:57.511409 15606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:57.512382 15606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:57.513552 15606 hybrid_clock.cc:648] HybridClock initialized: now 1790304777513534 us; error 28 us; skew 500 ppm
Sep 25 02:52:57 dist-test-slave-252r krb5kdc[15593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304777, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:57.516085 15606 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:52:57.516888 15606 webserver.cc:460] Webserver started at http://127.12.222.254:38333/ using document root <none> and password file <none>
I20260925 02:52:57.517386 15606 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:57.517443 15606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:57.517727 15606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:57.519601 15606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "161f8df3d3f648ae9c25d509e274eb92"
format_stamp: "Formatted at 2026-09-25 02:52:57 on dist-test-slave-252r"
server_key: "7896aa45b0b7b051464928aab8509124"
server_key_iv: "2819f689f64ef23276633ce676d4555a"
server_key_version: "encryptionkey@0"
I20260925 02:52:57.520160 15606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "161f8df3d3f648ae9c25d509e274eb92"
format_stamp: "Formatted at 2026-09-25 02:52:57 on dist-test-slave-252r"
server_key: "7896aa45b0b7b051464928aab8509124"
server_key_iv: "2819f689f64ef23276633ce676d4555a"
server_key_version: "encryptionkey@0"
I20260925 02:52:57.524202 15606 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 02:52:57.526516 15621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:57.527683 15606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:57.527809 15606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "161f8df3d3f648ae9c25d509e274eb92"
format_stamp: "Formatted at 2026-09-25 02:52:57 on dist-test-slave-252r"
server_key: "7896aa45b0b7b051464928aab8509124"
server_key_iv: "2819f689f64ef23276633ce676d4555a"
server_key_version: "encryptionkey@0"
I20260925 02:52:57.527921 15606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:57.547734 15606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:57.550290 15606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:57.550498 15606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:57.556912 15606 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:46199
I20260925 02:52:57.556938 15674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:46199 every 8 connection(s)
I20260925 02:52:57.557801 15606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:57.560319 15675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:57.565795 15675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92: Bootstrap starting.
I20260925 02:52:57.566315 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 15606
I20260925 02:52:57.566421 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:52:57.566699 13179 external_mini_cluster.cc:1428] Setting key 52bc806f9a9d9a7b6c630280927abb0e
I20260925 02:52:57.569900 15675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:57.570801 15675 log.cc:824] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:57.573390 15675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92: No bootstrap required, opened a new log
Sep 25 02:52:57 dist-test-slave-252r krb5kdc[15593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:57.578642 15675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161f8df3d3f648ae9c25d509e274eb92" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 46199 } }
I20260925 02:52:57.578871 15675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:57.578924 15675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 161f8df3d3f648ae9c25d509e274eb92, State: Initialized, Role: FOLLOWER
I20260925 02:52:57.579329 15675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161f8df3d3f648ae9c25d509e274eb92" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 46199 } }
I20260925 02:52:57.579597 15675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:57.579656 15675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:57.579738 15675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:57.580845 15675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161f8df3d3f648ae9c25d509e274eb92" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 46199 } }
I20260925 02:52:57.581204 15675 leader_election.cc:302] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 161f8df3d3f648ae9c25d509e274eb92; no voters:
I20260925 02:52:57.581518 15675 leader_election.cc:288] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:57.581591 15680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:57.581777 15680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92 [term 1 LEADER]: Becoming Leader. State: Replica: 161f8df3d3f648ae9c25d509e274eb92, State: Running, Role: LEADER
I20260925 02:52:57.582054 15680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161f8df3d3f648ae9c25d509e274eb92" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 46199 } }
I20260925 02:52:57.582708 15675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:57.583123 15680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 161f8df3d3f648ae9c25d509e274eb92. Latest consensus state: current_term: 1 leader_uuid: "161f8df3d3f648ae9c25d509e274eb92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161f8df3d3f648ae9c25d509e274eb92" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 46199 } } }
I20260925 02:52:57.583246 15680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:57.583118 15681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "161f8df3d3f648ae9c25d509e274eb92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161f8df3d3f648ae9c25d509e274eb92" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 46199 } } }
I20260925 02:52:57.583455 15681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:57.583770 15684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:57.586280 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:57.568334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50432 (local address 127.12.222.254:46199)
0925 02:52:57.568751 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:57.568768 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:57.568810 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:52:57.569620 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:57.569647 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:57.570100 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:57.570500 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:57.571653 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:57.573297 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:57.574682 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:57.575453 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:57.576033 (+ 580us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:57.578549 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:57.578555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:57.578570 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:57.578603 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:57.582536 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:57.583069 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:57.583075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:57.583080 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:57.583223 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:57.583569 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:57.583573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:57.583575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:57.584056 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:57.584226 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:57.585153 (+ 927us) server_negotiation.cc:299] Negotiation successful
0925 02:52:57.585416 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":137,"threads_started":1}
I20260925 02:52:57.586907 15684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:57.593779 15684 catalog_manager.cc:1347] Generated new cluster ID: b2d1194ba77d4d51abc27cb3b585d3ed
I20260925 02:52:57.593832 15684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:57.618079 15684 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:57.619460 15684 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:57.627307 15684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 161f8df3d3f648ae9c25d509e274eb92: Generated new TSK 0
I20260925 02:52:57.627954 15684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
I20260925 02:52:57.654778 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:46199
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:33515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:57.761793 15701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:57.762006 15701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:57.762058 15701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:57.765700 15701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:57.765749 15701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:57.765817 15701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:52:57.770609 15701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:33515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:46199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:57.771500 15701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:57.772400 15701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:57.781056 15706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:57.781138 15701 server_base.cc:1034] running on GCE node
W20260925 02:52:57.781036 15707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:57.781041 15709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:57.781804 15701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:57.782402 15701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:57.783587 15701 hybrid_clock.cc:648] HybridClock initialized: now 1790304777783572 us; error 31 us; skew 500 ppm
Sep 25 02:52:57 dist-test-slave-252r krb5kdc[15593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304777, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:57.786240 15701 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:52:57.787006 15701 webserver.cc:460] Webserver started at http://127.12.222.193:36651/ using document root <none> and password file <none>
I20260925 02:52:57.787452 15701 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:57.787554 15701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:57.787766 15701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:57.789753 15701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "9eea59420a1e487982af8cc34d518bd1"
format_stamp: "Formatted at 2026-09-25 02:52:57 on dist-test-slave-252r"
server_key: "cc2cb2089fc880ede7e89df260a20887"
server_key_iv: "e1995f3fce04b90eddbbc1bd8d75b635"
server_key_version: "encryptionkey@0"
I20260925 02:52:57.790230 15701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "9eea59420a1e487982af8cc34d518bd1"
format_stamp: "Formatted at 2026-09-25 02:52:57 on dist-test-slave-252r"
server_key: "cc2cb2089fc880ede7e89df260a20887"
server_key_iv: "e1995f3fce04b90eddbbc1bd8d75b635"
server_key_version: "encryptionkey@0"
I20260925 02:52:57.793486 15701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:52:57.795631 15716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:57.796765 15701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:57.796855 15701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "9eea59420a1e487982af8cc34d518bd1"
format_stamp: "Formatted at 2026-09-25 02:52:57 on dist-test-slave-252r"
server_key: "cc2cb2089fc880ede7e89df260a20887"
server_key_iv: "e1995f3fce04b90eddbbc1bd8d75b635"
server_key_version: "encryptionkey@0"
I20260925 02:52:57.796936 15701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:57.809746 15701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:57.812547 15701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:57.812693 15701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:57.813181 15701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:52:57.813869 15701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:57.813913 15701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:57.813957 15701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:57.813977 15701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:57.822495 15701 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:41813
I20260925 02:52:57.822530 15830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:41813 every 8 connection(s)
I20260925 02:52:57.823323 15701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:52:57.829795 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 15701
I20260925 02:52:57.829891 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:52:57.830142 13179 external_mini_cluster.cc:1428] Setting key e6069822b5e2aac7cdc2b7d84a8822ad
Sep 25 02:52:57 dist-test-slave-252r krb5kdc[15593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304777, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:57.836438 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:57.825028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:51729 (local address 127.12.222.254:46199)
0925 02:52:57.825187 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:57.825191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:57.826302 (+ 1111us) server_negotiation.cc:407] Connection header received
0925 02:52:57.827156 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:57.827159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:57.827213 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:57.827307 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:57.828329 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:57.828820 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:57.830034 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:57.830329 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:57.830461 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:57.832966 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:57.832970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:57.832972 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:57.832998 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:57.834365 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:57.834916 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:57.834919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:57.834920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:57.834993 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:57.835345 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:57.835347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:57.835348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:57.835509 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:57.835580 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:57.836182 (+ 602us) server_negotiation.cc:299] Negotiation successful
0925 02:52:57.836322 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 02:52:57.837594 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:57.825390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:46199 (local address 127.12.222.193:51729)
0925 02:52:57.826167 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:57.826205 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:57.826968 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:57.827465 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:57.827470 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:57.827762 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:57.828173 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:57.828185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:57.829011 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:57.829017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:57.829839 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:57.829910 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:57.830499 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:57.830503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:57.830702 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:57.831316 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:57.831337 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:57.832804 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:57.834475 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:57.834483 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:57.834497 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:57.834808 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:57.835086 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:57.835092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:57.835096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:57.835260 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:57.835605 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:57.835613 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:57.835922 (+ 309us) client_negotiation.cc:769] Sending connection context
0925 02:52:57.836138 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 02:52:57.836482 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":478,"threads_started":1}
I20260925 02:52:57.838747 15831 heartbeater.cc:347] Connected to a master server at 127.12.222.254:46199
I20260925 02:52:57.838979 15831 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:57.839423 15831 heartbeater.cc:511] Master 127.12.222.254:46199 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.222.194@KRBTEST.COM; defaulting to no policy
I20260925 02:52:57.840857 15638 ts_manager.cc:194] Registered new tserver with Master: 9eea59420a1e487982af8cc34d518bd1 (127.12.222.193:41813)
I20260925 02:52:57.842106 15638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:51729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
I20260925 02:52:57.856567 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:46199
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:33515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:57.955932 15836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:57.956176 15836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:57.956262 15836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:57.959824 15836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:57.959913 15836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:57.959990 15836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:52:57.964711 15836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:33515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:46199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:57.965580 15836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:57.966461 15836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:57.975060 15842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:57.975054 15844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:57.975054 15841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:57.975389 15836 server_base.cc:1034] running on GCE node
I20260925 02:52:57.975788 15836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:57.976310 15836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:57.977478 15836 hybrid_clock.cc:648] HybridClock initialized: now 1790304777977448 us; error 36 us; skew 500 ppm
Sep 25 02:52:57 dist-test-slave-252r krb5kdc[15593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304777, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:57.980118 15836 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:52:57.980962 15836 webserver.cc:460] Webserver started at http://127.12.222.194:42135/ using document root <none> and password file <none>
I20260925 02:52:57.981455 15836 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:57.981514 15836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:57.981729 15836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:57.983596 15836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "681be31d425f42bdb05309e0c06a4e8f"
format_stamp: "Formatted at 2026-09-25 02:52:57 on dist-test-slave-252r"
server_key: "dd2066324d1fff59e3e2684382546f82"
server_key_iv: "8ecb69ec2cc13af9082018cbcfdbc5df"
server_key_version: "encryptionkey@0"
I20260925 02:52:57.984273 15836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "681be31d425f42bdb05309e0c06a4e8f"
format_stamp: "Formatted at 2026-09-25 02:52:57 on dist-test-slave-252r"
server_key: "dd2066324d1fff59e3e2684382546f82"
server_key_iv: "8ecb69ec2cc13af9082018cbcfdbc5df"
server_key_version: "encryptionkey@0"
I20260925 02:52:57.987447 15836 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260925 02:52:57.989493 15851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:57.990646 15836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:52:57.990756 15836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "681be31d425f42bdb05309e0c06a4e8f"
format_stamp: "Formatted at 2026-09-25 02:52:57 on dist-test-slave-252r"
server_key: "dd2066324d1fff59e3e2684382546f82"
server_key_iv: "8ecb69ec2cc13af9082018cbcfdbc5df"
server_key_version: "encryptionkey@0"
I20260925 02:52:57.990849 15836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:57.998642 15836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:58.001748 15836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:58.001925 15836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:58.002461 15836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:52:58.003207 15836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:58.003252 15836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:58.003319 15836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:58.003356 15836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:58.011348 15836 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:40929
I20260925 02:52:58.011396 15965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:40929 every 8 connection(s)
I20260925 02:52:58.012144 15836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:52:58 dist-test-slave-252r krb5kdc[15593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304777, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:58.021718 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 15836
I20260925 02:52:58.021857 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:52:58.022224 13179 external_mini_cluster.cc:1428] Setting key f70a4c186735d573c9c84269a87e45a8
I20260925 02:52:58.024897 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:58.013868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:36377 (local address 127.12.222.254:46199)
0925 02:52:58.014000 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:58.014004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:58.014713 (+ 709us) server_negotiation.cc:407] Connection header received
0925 02:52:58.015943 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:58.015947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:58.016004 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:58.016118 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:58.017184 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:58.017662 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:58.018779 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:58.018984 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:58.019083 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:58.021221 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:58.021224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:58.021226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:58.021249 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:58.022685 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:58.023212 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:58.023215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:58.023217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:58.023297 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:58.023698 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:58.023701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:58.023703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:58.023886 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:58.023962 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:58.024589 (+ 627us) server_negotiation.cc:299] Negotiation successful
0925 02:52:58.024687 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:52:58.025853 15968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:58.014210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:46199 (local address 127.12.222.194:36377)
0925 02:52:58.014581 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:58.014618 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:58.015742 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:58.016310 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:58.016316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:58.016653 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:58.017043 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:58.017054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:58.017783 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:58.017785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:58.018640 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:58.018649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:58.019117 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:58.019121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:58.019268 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:58.019761 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:58.019774 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:58.021061 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:58.022798 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:58.022804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:58.022815 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:58.023116 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:58.023405 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:58.023409 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:58.023411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:58.023583 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:58.023996 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:58.024004 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:58.024308 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 02:52:58.024516 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 02:52:58.024819 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
I20260925 02:52:58.026930 15966 heartbeater.cc:347] Connected to a master server at 127.12.222.254:46199
I20260925 02:52:58.027144 15966 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:58.027545 15966 heartbeater.cc:511] Master 127.12.222.254:46199 requested a full tablet report, sending...
I20260925 02:52:58.028555 15638 ts_manager.cc:194] Registered new tserver with Master: 681be31d425f42bdb05309e0c06a4e8f (127.12.222.194:40929)
I20260925 02:52:58.029062 15638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:36377
WARNING: no policy specified for kudu/127.12.222.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
I20260925 02:52:58.046837 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:46199
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:33515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:58.148617 15971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:58.148835 15971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:58.148907 15971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:58.152716 15971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:58.152783 15971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:58.152866 15971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:52:58.157498 15971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:33515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:46199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:58.158388 15971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:58.159267 15971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:58.168045 15976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:58.168062 15977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:58.168062 15979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:58.168416 15971 server_base.cc:1034] running on GCE node
I20260925 02:52:58.168872 15971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:58.169433 15971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:58.170619 15971 hybrid_clock.cc:648] HybridClock initialized: now 1790304778170603 us; error 31 us; skew 500 ppm
Sep 25 02:52:58 dist-test-slave-252r krb5kdc[15593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304778, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:58.173302 15971 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:52:58.174163 15971 webserver.cc:460] Webserver started at http://127.12.222.195:39379/ using document root <none> and password file <none>
I20260925 02:52:58.174679 15971 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:58.174767 15971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:58.175009 15971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:58.176944 15971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "22d994606f8843d8a12e29f6a20b4e5a"
format_stamp: "Formatted at 2026-09-25 02:52:58 on dist-test-slave-252r"
server_key: "b085cf8198fb117af63925aa78248db0"
server_key_iv: "143d6063a7f84267953767279c6cd31d"
server_key_version: "encryptionkey@0"
I20260925 02:52:58.177479 15971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "22d994606f8843d8a12e29f6a20b4e5a"
format_stamp: "Formatted at 2026-09-25 02:52:58 on dist-test-slave-252r"
server_key: "b085cf8198fb117af63925aa78248db0"
server_key_iv: "143d6063a7f84267953767279c6cd31d"
server_key_version: "encryptionkey@0"
I20260925 02:52:58.180836 15971 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:52:58.183120 15986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:58.184443 15971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:52:58.184574 15971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "22d994606f8843d8a12e29f6a20b4e5a"
format_stamp: "Formatted at 2026-09-25 02:52:58 on dist-test-slave-252r"
server_key: "b085cf8198fb117af63925aa78248db0"
server_key_iv: "143d6063a7f84267953767279c6cd31d"
server_key_version: "encryptionkey@0"
I20260925 02:52:58.184695 15971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:58.204916 15971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:58.207990 15971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:58.208170 15971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:58.208719 15971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:52:58.209470 15971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:58.209534 15971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:58.209597 15971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:58.209645 15971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:58.217792 15971 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:41209
I20260925 02:52:58.217824 16100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:41209 every 8 connection(s)
I20260925 02:52:58.218523 15971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:52:58.222033 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 15971
I20260925 02:52:58.222144 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:52:58.222364 13179 external_mini_cluster.cc:1428] Setting key 9aafe5abb2d13b50dc130f80520ea79a
Sep 25 02:52:58 dist-test-slave-252r krb5kdc[15593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304778, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:58.230414 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:58.220029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:50741 (local address 127.12.222.254:46199)
0925 02:52:58.220176 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:58.220180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:58.220926 (+ 746us) server_negotiation.cc:407] Connection header received
0925 02:52:58.222066 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:58.222069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:58.222120 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:58.222218 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:58.223463 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:58.224124 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:58.225265 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:58.225458 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:58.225532 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:58.227487 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:58.227490 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:58.227492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:58.227513 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:58.228833 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:58.229256 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:58.229258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:58.229259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:58.229325 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:58.229592 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:58.229593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:58.229594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:58.229737 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:58.229805 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:58.230174 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 02:52:58.230272 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:52:58.231097 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:58.220386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:46199 (local address 127.12.222.195:50741)
0925 02:52:58.220767 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:58.220803 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:58.221876 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:58.222371 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:58.222378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:58.222777 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:58.223324 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:58.223339 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:58.224282 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:58.224286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:58.225141 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:58.225151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:58.225573 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:58.225575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:58.225697 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:58.226118 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:58.226132 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:58.227341 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:58.228935 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:58.228939 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:58.228948 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:58.229168 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:58.229397 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:58.229399 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:58.229400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:58.229519 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:58.229816 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:58.229819 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:58.229997 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 02:52:58.230130 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 02:52:58.230329 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":89,"threads_started":1}
I20260925 02:52:58.232221 16101 heartbeater.cc:347] Connected to a master server at 127.12.222.254:46199
I20260925 02:52:58.232424 16101 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:58.232795 16101 heartbeater.cc:511] Master 127.12.222.254:46199 requested a full tablet report, sending...
I20260925 02:52:58.233700 15638 ts_manager.cc:194] Registered new tserver with Master: 22d994606f8843d8a12e29f6a20b4e5a (127.12.222.195:41209)
I20260925 02:52:58.234218 15638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:50741
I20260925 02:52:58.235889 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:52:58.247416 15678 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:58.246250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.12.222.254:46199)
0925 02:52:58.246362 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:58.246365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:58.246556 (+ 191us) server_negotiation.cc:407] Connection header received
0925 02:52:58.246648 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:58.246650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:58.246683 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:58.246745 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:58.247296 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50444: BlockingRecv error: recv got EOF from 127.0.0.1:50444 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:52:58.248489 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 15701
I20260925 02:52:58.254009 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 15836
I20260925 02:52:58.259385 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 15971
I20260925 02:52:58.266722 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 15606
2026-09-25T02:52:58Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1963 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:58 dist-test-slave-252r krb5kdc[16113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:58 dist-test-slave-252r krb5kdc[16113](info): set up 2 sockets
Sep 25 02:52:58 dist-test-slave-252r krb5kdc[16113](info): commencing operation
krb5kdc: starting...
W20260925 02:52:59.231649 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:52:59 dist-test-slave-252r krb5kdc[16113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:52:59.342486 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:41099
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:41099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:59.442076 16126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:59.442279 16126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:59.442312 16126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:59.446000 16126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:59.446040 16126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:59.446058 16126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:59.446074 16126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:59.446087 16126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:59.450562 16126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:41099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:41099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:59.451336 16126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:59.452172 16126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:59.459604 16134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:59.459822 16126 server_base.cc:1034] running on GCE node
W20260925 02:52:59.459623 16132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:59.459612 16131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:59.460470 16126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:59.461458 16126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:59.462610 16126 hybrid_clock.cc:648] HybridClock initialized: now 1790304779462586 us; error 35 us; skew 500 ppm
Sep 25 02:52:59 dist-test-slave-252r krb5kdc[16113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304779, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:59.473167 16126 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:52:59.474032 16126 webserver.cc:460] Webserver started at http://127.12.222.254:46377/ using document root <none> and password file <none>
I20260925 02:52:59.474552 16126 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:59.474606 16126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:59.474813 16126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:59.476702 16126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "fba3c7fd846e4bc2924b5bd6562ce841"
format_stamp: "Formatted at 2026-09-25 02:52:59 on dist-test-slave-252r"
server_key: "f3b05fe908b48eaa38e8fc7cac9db6cf"
server_key_iv: "eda25c58a7a6d13c08c03d3dc3e3d59d"
server_key_version: "encryptionkey@0"
I20260925 02:52:59.477224 16126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "fba3c7fd846e4bc2924b5bd6562ce841"
format_stamp: "Formatted at 2026-09-25 02:52:59 on dist-test-slave-252r"
server_key: "f3b05fe908b48eaa38e8fc7cac9db6cf"
server_key_iv: "eda25c58a7a6d13c08c03d3dc3e3d59d"
server_key_version: "encryptionkey@0"
I20260925 02:52:59.480475 16126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:52:59.482594 16141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:59.483762 16126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:59.483911 16126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "fba3c7fd846e4bc2924b5bd6562ce841"
format_stamp: "Formatted at 2026-09-25 02:52:59 on dist-test-slave-252r"
server_key: "f3b05fe908b48eaa38e8fc7cac9db6cf"
server_key_iv: "eda25c58a7a6d13c08c03d3dc3e3d59d"
server_key_version: "encryptionkey@0"
I20260925 02:52:59.483996 16126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:59.505218 16126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:59.507979 16126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:59.508150 16126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:59.514374 16126 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:41099
I20260925 02:52:59.514647 16194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:41099 every 8 connection(s)
I20260925 02:52:59.515201 16126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:59.517584 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 16126
I20260925 02:52:59.517679 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:52:59.517685 16195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:59.517941 13179 external_mini_cluster.cc:1428] Setting key d99a75c3229ea48012c2d65686b79ce5
I20260925 02:52:59.524060 16195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841: Bootstrap starting.
Sep 25 02:52:59 dist-test-slave-252r krb5kdc[16113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:52:59.526544 16195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:59.527340 16195 log.cc:824] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:59.529218 16195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841: No bootstrap required, opened a new log
I20260925 02:52:59.531320 16198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:59.519053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.12.222.254:41099)
0925 02:52:59.519402 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:59.519415 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:59.519446 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:52:59.520009 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:59.520025 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:59.520371 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:59.520618 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:59.521528 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:59.522417 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:59.523520 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:59.523943 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:59.524051 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:59.526712 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:59.526716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:59.526728 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:59.526754 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:59.528664 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:59.529098 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:59.529104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:59.529109 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:59.529225 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:59.529529 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:59.529532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:59.529534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:59.529897 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:59.530088 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:59.530374 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 02:52:59.530634 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":111,"threads_started":1}
I20260925 02:52:59.532711 16195 raft_consensus.cc:348] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba3c7fd846e4bc2924b5bd6562ce841" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 41099 } }
I20260925 02:52:59.533022 16195 raft_consensus.cc:374] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:59.533113 16195 raft_consensus.cc:729] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba3c7fd846e4bc2924b5bd6562ce841, State: Initialized, Role: FOLLOWER
I20260925 02:52:59.533535 16195 consensus_queue.cc:260] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba3c7fd846e4bc2924b5bd6562ce841" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 41099 } }
I20260925 02:52:59.533674 16195 raft_consensus.cc:388] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:59.533752 16195 raft_consensus.cc:482] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:59.533845 16195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:59.533954 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:59.534869 16195 raft_consensus.cc:504] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba3c7fd846e4bc2924b5bd6562ce841" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 41099 } }
I20260925 02:52:59.535241 16195 leader_election.cc:302] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fba3c7fd846e4bc2924b5bd6562ce841; no voters:
I20260925 02:52:59.535552 16195 leader_election.cc:288] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:59.535670 16200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:59.535910 16200 raft_consensus.cc:686] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841 [term 1 LEADER]: Becoming Leader. State: Replica: fba3c7fd846e4bc2924b5bd6562ce841, State: Running, Role: LEADER
I20260925 02:52:59.536340 16200 consensus_queue.cc:237] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba3c7fd846e4bc2924b5bd6562ce841" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 41099 } }
I20260925 02:52:59.536743 16195 sys_catalog.cc:564] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:59.537563 16200 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fba3c7fd846e4bc2924b5bd6562ce841. Latest consensus state: current_term: 1 leader_uuid: "fba3c7fd846e4bc2924b5bd6562ce841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba3c7fd846e4bc2924b5bd6562ce841" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 41099 } } }
I20260925 02:52:59.537744 16200 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:59.538122 16210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:59.539304 16201 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fba3c7fd846e4bc2924b5bd6562ce841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba3c7fd846e4bc2924b5bd6562ce841" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 41099 } } }
I20260925 02:52:59.539479 16201 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:59.540934 16210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:59.546322 16210 catalog_manager.cc:1347] Generated new cluster ID: 32e3a796c93d4a89b8209864f1fbfbf4
I20260925 02:52:59.546397 16210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:59.557055 16198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:59.551904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.12.222.254:41099)
0925 02:52:59.552050 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:59.552054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:59.552297 (+ 243us) server_negotiation.cc:407] Connection header received
0925 02:52:59.552424 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:59.552429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:59.552476 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:59.552554 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:59.553085 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:59.553844 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:59.554866 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:59.555207 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:59.555301 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:59.556112 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:59.556116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:59.556118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:59.556618 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:59.556648 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:59.556666 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:59.556749 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:59.556868 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 02:52:59.556917 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:52:59.558863 16210 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:59.559615 16210 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:59.572079 16210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fba3c7fd846e4bc2924b5bd6562ce841: Generated new TSK 0
I20260925 02:52:59.572644 16210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:59.578943 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 16126
2026-09-25T02:52:59Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1310 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:59 dist-test-slave-252r krb5kdc[16228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:59 dist-test-slave-252r krb5kdc[16228](info): set up 2 sockets
Sep 25 02:52:59 dist-test-slave-252r krb5kdc[16228](info): commencing operation
krb5kdc: starting...
W20260925 02:53:00.563995 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:53:00 dist-test-slave-252r krb5kdc[16228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:53:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:53:00.673051 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:35979
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:37601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:35979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:00.774005 16241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:00.774237 16241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:00.774276 16241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:00.778028 16241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:00.778074 16241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:00.778090 16241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:00.778105 16241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:00.778118 16241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:00.782667 16241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:37601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:35979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:35979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:00.783486 16241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:00.784448 16241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:53:00.791949 16241 server_base.cc:1034] running on GCE node
W20260925 02:53:00.791980 16249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:00.791916 16247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:00.791917 16246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:00.792749 16241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:00.793816 16241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:00.794972 16241 hybrid_clock.cc:648] HybridClock initialized: now 1790304780794950 us; error 37 us; skew 500 ppm
Sep 25 02:53:00 dist-test-slave-252r krb5kdc[16228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304780, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:00.797587 16241 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:53:00.798470 16241 webserver.cc:460] Webserver started at http://127.12.222.254:40841/ using document root <none> and password file <none>
I20260925 02:53:00.798908 16241 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:00.798967 16241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:00.799109 16241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:00.800945 16241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "c74213de5c8943ad900e4a450629b2ad"
format_stamp: "Formatted at 2026-09-25 02:53:00 on dist-test-slave-252r"
server_key: "bc152a6859f127905f034bdcdcf9ebd7"
server_key_iv: "9736bb12acf1e1aa0e8689b9af18b5d5"
server_key_version: "encryptionkey@0"
I20260925 02:53:00.801419 16241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "c74213de5c8943ad900e4a450629b2ad"
format_stamp: "Formatted at 2026-09-25 02:53:00 on dist-test-slave-252r"
server_key: "bc152a6859f127905f034bdcdcf9ebd7"
server_key_iv: "9736bb12acf1e1aa0e8689b9af18b5d5"
server_key_version: "encryptionkey@0"
I20260925 02:53:00.804776 16241 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:53:00.807046 16256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:00.808192 16241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:53:00.808336 16241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "c74213de5c8943ad900e4a450629b2ad"
format_stamp: "Formatted at 2026-09-25 02:53:00 on dist-test-slave-252r"
server_key: "bc152a6859f127905f034bdcdcf9ebd7"
server_key_iv: "9736bb12acf1e1aa0e8689b9af18b5d5"
server_key_version: "encryptionkey@0"
I20260925 02:53:00.808420 16241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:00.823246 16241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:00.825714 16241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:00.825847 16241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:00.832036 16241 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:35979
I20260925 02:53:00.832365 16308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:35979 every 8 connection(s)
I20260925 02:53:00.832942 16241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:00.835397 16309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:00.837826 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 16241
I20260925 02:53:00.837937 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:53:00.838205 13179 external_mini_cluster.cc:1428] Setting key 963f004273db0dba752961f6f6d3c1fd
I20260925 02:53:00.840966 16309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad: Bootstrap starting.
I20260925 02:53:00.842963 16309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:00.843753 16309 log.cc:824] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:00.845592 16309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad: No bootstrap required, opened a new log
Sep 25 02:53:00 dist-test-slave-252r krb5kdc[16228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:00.848681 16309 raft_consensus.cc:348] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c74213de5c8943ad900e4a450629b2ad" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 35979 } }
I20260925 02:53:00.848878 16309 raft_consensus.cc:374] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:00.848937 16309 raft_consensus.cc:729] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74213de5c8943ad900e4a450629b2ad, State: Initialized, Role: FOLLOWER
I20260925 02:53:00.849354 16309 consensus_queue.cc:260] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c74213de5c8943ad900e4a450629b2ad" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 35979 } }
I20260925 02:53:00.849490 16309 raft_consensus.cc:388] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:00.849556 16309 raft_consensus.cc:482] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:00.849659 16309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:00.850365 16309 raft_consensus.cc:504] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c74213de5c8943ad900e4a450629b2ad" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 35979 } }
I20260925 02:53:00.850677 16309 leader_election.cc:302] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c74213de5c8943ad900e4a450629b2ad; no voters:
I20260925 02:53:00.850981 16309 leader_election.cc:288] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:53:00.851045 16314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:00.851332 16314 raft_consensus.cc:686] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad [term 1 LEADER]: Becoming Leader. State: Replica: c74213de5c8943ad900e4a450629b2ad, State: Running, Role: LEADER
I20260925 02:53:00.851640 16314 consensus_queue.cc:237] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c74213de5c8943ad900e4a450629b2ad" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 35979 } }
I20260925 02:53:00.852164 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:00.839483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60036 (local address 127.12.222.254:35979)
0925 02:53:00.839848 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:00.839861 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:00.839892 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:53:00.840544 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:00.840562 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:00.840911 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:00.841264 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:00.842033 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:00.842975 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:00.843896 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:00.844412 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:00.844569 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:00.847057 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:00.847062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:00.847073 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:00.847095 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:00.849277 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:00.849751 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:00.849757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:00.849763 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:00.849878 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:00.850149 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:00.850152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:00.850154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:00.850546 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:00.850724 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:00.850994 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 02:53:00.851291 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":117,"threads_started":1}
I20260925 02:53:00.852191 16309 sys_catalog.cc:564] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:00.852744 16314 sys_catalog.cc:455] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader c74213de5c8943ad900e4a450629b2ad. Latest consensus state: current_term: 1 leader_uuid: "c74213de5c8943ad900e4a450629b2ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c74213de5c8943ad900e4a450629b2ad" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 35979 } } }
I20260925 02:53:00.852912 16314 sys_catalog.cc:458] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:00.853114 16315 sys_catalog.cc:455] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c74213de5c8943ad900e4a450629b2ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c74213de5c8943ad900e4a450629b2ad" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 35979 } } }
I20260925 02:53:00.853252 16315 sys_catalog.cc:458] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:00.854250 16321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:00.856741 16321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:00.861392 16321 catalog_manager.cc:1347] Generated new cluster ID: 86f2c7190440453382e231ca43b57c7b
I20260925 02:53:00.861445 16321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:00.876276 16321 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:00.877074 16321 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:00.881510 16321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c74213de5c8943ad900e4a450629b2ad: Generated new TSK 0
I20260925 02:53:00.882076 16321 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
I20260925 02:53:00.908946 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:35979
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:37601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:01.008955 16336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:01.009166 16336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:01.009235 16336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:01.013067 16336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:01.013137 16336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:01.013247 16336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:53:01.018157 16336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:37601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:35979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:01.019076 16336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:01.019986 16336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:01.028707 16341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:01.028695 16342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:01.028991 16336 server_base.cc:1034] running on GCE node
W20260925 02:53:01.028697 16344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:01.029448 16336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:01.029978 16336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:01.031149 16336 hybrid_clock.cc:648] HybridClock initialized: now 1790304781031133 us; error 28 us; skew 500 ppm
Sep 25 02:53:01 dist-test-slave-252r krb5kdc[16228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304781, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:01.033593 16336 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:53:01.034339 16336 webserver.cc:460] Webserver started at http://127.12.222.193:41547/ using document root <none> and password file <none>
I20260925 02:53:01.034801 16336 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:01.034885 16336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:01.035094 16336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:01.036863 16336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "5176664ac4e9451ca25a3f519850dcad"
format_stamp: "Formatted at 2026-09-25 02:53:01 on dist-test-slave-252r"
server_key: "d1288e266c02369d03c84401c0d3dcd7"
server_key_iv: "d20deaa88ff95f09ee4d7b1688243f96"
server_key_version: "encryptionkey@0"
I20260925 02:53:01.037453 16336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "5176664ac4e9451ca25a3f519850dcad"
format_stamp: "Formatted at 2026-09-25 02:53:01 on dist-test-slave-252r"
server_key: "d1288e266c02369d03c84401c0d3dcd7"
server_key_iv: "d20deaa88ff95f09ee4d7b1688243f96"
server_key_version: "encryptionkey@0"
I20260925 02:53:01.040807 16336 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:53:01.042932 16351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:01.044139 16336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:01.044296 16336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "5176664ac4e9451ca25a3f519850dcad"
format_stamp: "Formatted at 2026-09-25 02:53:01 on dist-test-slave-252r"
server_key: "d1288e266c02369d03c84401c0d3dcd7"
server_key_iv: "d20deaa88ff95f09ee4d7b1688243f96"
server_key_version: "encryptionkey@0"
I20260925 02:53:01.044414 16336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:01.060467 16336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:01.062865 16336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:01.063036 16336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:01.063561 16336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:01.064343 16336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:01.064422 16336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:01.064484 16336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:01.064524 16336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:01.072217 16336 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:38863
I20260925 02:53:01.072270 16464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:38863 every 8 connection(s)
I20260925 02:53:01.072979 16336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:53:01.073828 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 16336
I20260925 02:53:01.073961 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:53:01.074275 13179 external_mini_cluster.cc:1428] Setting key fb02a40c46281cb729e26e2beaf9f6fd
Sep 25 02:53:01 dist-test-slave-252r krb5kdc[16228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304781, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.222.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.194@KRBTEST.COM" created.
I20260925 02:53:01.087699 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:01.074728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:56061 (local address 127.12.222.254:35979)
0925 02:53:01.074854 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:01.074859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:01.075647 (+ 788us) server_negotiation.cc:407] Connection header received
0925 02:53:01.077057 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:01.077060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:01.077111 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:01.077198 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:01.078830 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:01.079328 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:01.081128 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:01.081359 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:01.081440 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:01.084243 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:01.084247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:01.084250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:01.084276 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:01.085673 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:01.086108 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:01.086110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:01.086112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:01.086183 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:01.086483 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:01.086485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:01.086486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:01.086688 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:01.086797 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:01.087322 (+ 525us) server_negotiation.cc:299] Negotiation successful
0925 02:53:01.087460 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:53:01.088181 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:01.075060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:35979 (local address 127.12.222.193:56061)
0925 02:53:01.075512 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:01.075548 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:01.076857 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:01.077502 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:01.077514 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:01.078014 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:01.078681 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:01.078701 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:01.079456 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:01.079462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:01.080996 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:01.081011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:01.081518 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:01.081525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:01.081761 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:01.082329 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:01.082352 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:01.084037 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:01.085775 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:01.085779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:01.085789 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:01.086024 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:01.086263 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:01.086265 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:01.086266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:01.086382 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:01.086817 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:01.086825 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:01.087091 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 02:53:01.087244 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 02:53:01.087450 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":98,"threads_started":1}
I20260925 02:53:01.089334 16465 heartbeater.cc:347] Connected to a master server at 127.12.222.254:35979
I20260925 02:53:01.089546 16465 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:01.089951 16465 heartbeater.cc:511] Master 127.12.222.254:35979 requested a full tablet report, sending...
I20260925 02:53:01.091300 16273 ts_manager.cc:194] Registered new tserver with Master: 5176664ac4e9451ca25a3f519850dcad (127.12.222.193:38863)
I20260925 02:53:01.092355 16273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:56061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
I20260925 02:53:01.101169 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:35979
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:37601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:01.201454 16470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:01.201707 16470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:01.201777 16470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:01.205550 16470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:01.205642 16470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:01.205725 16470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:53:01.210856 16470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:37601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:35979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:01.211767 16470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:01.212738 16470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:01.221763 16478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:01.221782 16475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:01.221845 16470 server_base.cc:1034] running on GCE node
W20260925 02:53:01.222033 16476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:01.222398 16470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:01.222997 16470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:01.224169 16470 hybrid_clock.cc:648] HybridClock initialized: now 1790304781224139 us; error 43 us; skew 500 ppm
Sep 25 02:53:01 dist-test-slave-252r krb5kdc[16228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304781, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:01.226689 16470 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:53:01.227455 16470 webserver.cc:460] Webserver started at http://127.12.222.194:33345/ using document root <none> and password file <none>
I20260925 02:53:01.227977 16470 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:01.228063 16470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:01.228331 16470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:01.230155 16470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "f8649db7129f49c58e78e525197c28f1"
format_stamp: "Formatted at 2026-09-25 02:53:01 on dist-test-slave-252r"
server_key: "beea19fe67a873588cae42a2423337a8"
server_key_iv: "faaf4cca6772b22ec11fe91572421521"
server_key_version: "encryptionkey@0"
I20260925 02:53:01.230664 16470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "f8649db7129f49c58e78e525197c28f1"
format_stamp: "Formatted at 2026-09-25 02:53:01 on dist-test-slave-252r"
server_key: "beea19fe67a873588cae42a2423337a8"
server_key_iv: "faaf4cca6772b22ec11fe91572421521"
server_key_version: "encryptionkey@0"
I20260925 02:53:01.233942 16470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:53:01.236143 16485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:01.237257 16470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:01.237396 16470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "f8649db7129f49c58e78e525197c28f1"
format_stamp: "Formatted at 2026-09-25 02:53:01 on dist-test-slave-252r"
server_key: "beea19fe67a873588cae42a2423337a8"
server_key_iv: "faaf4cca6772b22ec11fe91572421521"
server_key_version: "encryptionkey@0"
I20260925 02:53:01.237504 16470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:01.247264 16470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:01.250391 16470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:01.250581 16470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:01.251086 16470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:01.251844 16470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:01.251911 16470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:01.251962 16470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:01.252012 16470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:01.260376 16470 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:37449
I20260925 02:53:01.260417 16600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:37449 every 8 connection(s)
I20260925 02:53:01.261122 16470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:53:01.266156 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 16470
I20260925 02:53:01.266335 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:53:01.266592 13179 external_mini_cluster.cc:1428] Setting key 94c033d44d825972a684688868191d82
Sep 25 02:53:01 dist-test-slave-252r krb5kdc[16228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304781, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:01.274614 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:01.262650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:46433 (local address 127.12.222.254:35979)
0925 02:53:01.262795 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:01.262798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:01.263566 (+ 768us) server_negotiation.cc:407] Connection header received
0925 02:53:01.264922 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:01.264925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:01.264989 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:01.265084 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:01.266322 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:01.266998 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:01.268421 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:01.268628 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:01.268707 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:01.271158 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:01.271162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:01.271164 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:01.271193 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:01.272647 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:01.273136 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:01.273141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:01.273142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:01.273221 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:01.273547 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:01.273551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:01.273553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:01.273718 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:01.273821 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:01.274269 (+ 448us) server_negotiation.cc:299] Negotiation successful
0925 02:53:01.274400 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:53:01.275306 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:01.262987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:35979 (local address 127.12.222.194:46433)
0925 02:53:01.263412 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:01.263460 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:01.264729 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:01.265239 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:01.265244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:01.265545 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:01.266027 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:01.266048 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:01.267170 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:01.267173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:01.268303 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:01.268316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:01.268775 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:01.268782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:01.269006 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:01.269551 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:01.269577 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:01.270996 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:01.272762 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:01.272768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:01.272780 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:01.273032 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:01.273311 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:01.273315 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:01.273317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:01.273451 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:01.273821 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:01.273828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:01.274052 (+ 224us) client_negotiation.cc:769] Sending connection context
0925 02:53:01.274227 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 02:53:01.274565 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.12.222.195@KRBTEST.COM; defaulting to no policy
I20260925 02:53:01.276505 16601 heartbeater.cc:347] Connected to a master server at 127.12.222.254:35979
I20260925 02:53:01.276747 16601 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:01.277312 16601 heartbeater.cc:511] Master 127.12.222.254:35979 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.195@KRBTEST.COM" created.
I20260925 02:53:01.278391 16273 ts_manager.cc:194] Registered new tserver with Master: f8649db7129f49c58e78e525197c28f1 (127.12.222.194:37449)
I20260925 02:53:01.278935 16273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:46433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
I20260925 02:53:01.292865 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:35979
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:37601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:01.398018 16606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:01.398219 16606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:01.398257 16606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:01.401906 16606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:01.401957 16606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:01.402010 16606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:53:01.406831 16606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:37601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:35979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:01.407763 16606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:01.408607 16606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:01.417132 16612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:01.417153 16611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:01.417156 16614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:01.417419 16606 server_base.cc:1034] running on GCE node
I20260925 02:53:01.417851 16606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:01.418473 16606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:01.419622 16606 hybrid_clock.cc:648] HybridClock initialized: now 1790304781419598 us; error 36 us; skew 500 ppm
Sep 25 02:53:01 dist-test-slave-252r krb5kdc[16228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304781, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:01.422051 16606 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:53:01.422734 16606 webserver.cc:460] Webserver started at http://127.12.222.195:35265/ using document root <none> and password file <none>
I20260925 02:53:01.423172 16606 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:01.423230 16606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:01.423372 16606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:01.425165 16606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "be259d850deb4f7eaa2b977e618d8c9e"
format_stamp: "Formatted at 2026-09-25 02:53:01 on dist-test-slave-252r"
server_key: "6d7d3ff040f6b4a82afae13b56559e54"
server_key_iv: "21ff06eaa09d18cb48c7cbe8ba876563"
server_key_version: "encryptionkey@0"
I20260925 02:53:01.425731 16606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "be259d850deb4f7eaa2b977e618d8c9e"
format_stamp: "Formatted at 2026-09-25 02:53:01 on dist-test-slave-252r"
server_key: "6d7d3ff040f6b4a82afae13b56559e54"
server_key_iv: "21ff06eaa09d18cb48c7cbe8ba876563"
server_key_version: "encryptionkey@0"
I20260925 02:53:01.429067 16606 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:53:01.431095 16621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:01.432156 16606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:01.432299 16606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "be259d850deb4f7eaa2b977e618d8c9e"
format_stamp: "Formatted at 2026-09-25 02:53:01 on dist-test-slave-252r"
server_key: "6d7d3ff040f6b4a82afae13b56559e54"
server_key_iv: "21ff06eaa09d18cb48c7cbe8ba876563"
server_key_version: "encryptionkey@0"
I20260925 02:53:01.432403 16606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:01.442241 16606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:01.444772 16606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:01.444955 16606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:01.445495 16606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:01.446226 16606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:01.446295 16606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:01.446357 16606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:01.446405 16606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:01.455142 16606 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:43969
I20260925 02:53:01.455168 16735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:43969 every 8 connection(s)
I20260925 02:53:01.455960 16606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:53:01.457682 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 16606
I20260925 02:53:01.457813 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:53:01.458087 13179 external_mini_cluster.cc:1428] Setting key 475715da6adc9e8200d0cb117c7fb47e
Sep 25 02:53:01 dist-test-slave-252r krb5kdc[16228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304781, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:01.469240 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:01.457718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:39199 (local address 127.12.222.254:35979)
0925 02:53:01.457895 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:01.457899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:01.458728 (+ 829us) server_negotiation.cc:407] Connection header received
0925 02:53:01.460397 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:01.460401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:01.460467 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:01.460536 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:01.461613 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:01.462124 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:01.463171 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:01.463396 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:01.463478 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:01.466025 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:01.466028 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:01.466030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:01.466053 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:01.467248 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:01.467731 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:01.467733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:01.467734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:01.467803 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:01.468095 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:01.468097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:01.468098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:01.468315 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:01.468392 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:01.469028 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 02:53:01.469122 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 02:53:01.469980 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:01.458089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:35979 (local address 127.12.222.195:39199)
0925 02:53:01.458573 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:01.458625 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:01.460152 (+ 1527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:01.460697 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:01.460703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:01.461049 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:01.461490 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:01.461502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:01.462227 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:01.462229 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:01.463066 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:01.463075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:01.463677 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:01.463681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:01.463824 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:01.464433 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:01.464449 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:01.465857 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:01.467372 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:01.467377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:01.467388 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:01.467644 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:01.467881 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:01.467884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:01.467885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:01.468017 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:01.468406 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:01.468411 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:01.468616 (+ 205us) client_negotiation.cc:769] Sending connection context
0925 02:53:01.468981 (+ 365us) client_negotiation.cc:240] Negotiation successful
0925 02:53:01.469214 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":106,"threads_started":1}
I20260925 02:53:01.471211 16736 heartbeater.cc:347] Connected to a master server at 127.12.222.254:35979
I20260925 02:53:01.471487 16736 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:01.471995 16736 heartbeater.cc:511] Master 127.12.222.254:35979 requested a full tablet report, sending...
I20260925 02:53:01.473043 16273 ts_manager.cc:194] Registered new tserver with Master: be259d850deb4f7eaa2b977e618d8c9e (127.12.222.195:43969)
I20260925 02:53:01.473614 16273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:39199
I20260925 02:53:01.483560 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:53:01 dist-test-slave-252r krb5kdc[16228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304781, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 02:53:01 dist-test-slave-252r krb5kdc[16228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304781, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:01.506585 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:01.498996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60038 (local address 127.12.222.254:35979)
0925 02:53:01.499110 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:01.499114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:01.499326 (+ 212us) server_negotiation.cc:407] Connection header received
0925 02:53:01.499424 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:01.499426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:01.499469 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:01.499533 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:01.500186 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:01.500727 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:01.501534 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:01.501724 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:01.501793 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:01.503900 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:01.503903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:01.503905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:01.503926 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:01.505125 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:01.505514 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:01.505516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:01.505518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:01.505583 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:01.505843 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:01.505845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:01.505846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:01.506022 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:01.506117 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:01.506299 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 02:53:01.506377 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 02:53:01.506955 16273 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60038
I20260925 02:53:01.507740 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 16336
I20260925 02:53:01.513563 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 16470
I20260925 02:53:01.518968 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 16606
I20260925 02:53:01.524206 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 16241
2026-09-25T02:53:01Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1947 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:53:01 dist-test-slave-252r krb5kdc[16748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:53:01 dist-test-slave-252r krb5kdc[16748](info): set up 2 sockets
Sep 25 02:53:01 dist-test-slave-252r krb5kdc[16748](info): commencing operation
krb5kdc: starting...
W20260925 02:53:02.516306 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:53:02 dist-test-slave-252r krb5kdc[16748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:53:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:53:02.621506 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:38151
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:38151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:02.724548 16761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:02.724802 16761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:02.724879 16761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:02.728431 16761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:02.728493 16761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:02.728528 16761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:02.728575 16761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:02.728611 16761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:02.733201 16761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:38151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:38151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:02.734067 16761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:02.734956 16761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:02.742336 16767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:02.742336 16766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:02.742383 16769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:02.742465 16761 server_base.cc:1034] running on GCE node
I20260925 02:53:02.743055 16761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:02.743999 16761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:02.745189 16761 hybrid_clock.cc:648] HybridClock initialized: now 1790304782745150 us; error 50 us; skew 500 ppm
Sep 25 02:53:02 dist-test-slave-252r krb5kdc[16748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304782, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:02.747605 16761 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:53:02.748375 16761 webserver.cc:460] Webserver started at http://127.12.222.254:33335/ using document root <none> and password file <none>
I20260925 02:53:02.748831 16761 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:02.748881 16761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:02.749020 16761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:02.750751 16761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "bef4eec7ff47429c92e568cdbe618d3c"
format_stamp: "Formatted at 2026-09-25 02:53:02 on dist-test-slave-252r"
server_key: "2a36778c4e9fa2c5b6dde8f1d04c2760"
server_key_iv: "25c104b644e00e4bab68ea5293f0d587"
server_key_version: "encryptionkey@0"
I20260925 02:53:02.751222 16761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "bef4eec7ff47429c92e568cdbe618d3c"
format_stamp: "Formatted at 2026-09-25 02:53:02 on dist-test-slave-252r"
server_key: "2a36778c4e9fa2c5b6dde8f1d04c2760"
server_key_iv: "25c104b644e00e4bab68ea5293f0d587"
server_key_version: "encryptionkey@0"
I20260925 02:53:02.754496 16761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 02:53:02.756623 16776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:02.757782 16761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:53:02.757896 16761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "bef4eec7ff47429c92e568cdbe618d3c"
format_stamp: "Formatted at 2026-09-25 02:53:02 on dist-test-slave-252r"
server_key: "2a36778c4e9fa2c5b6dde8f1d04c2760"
server_key_iv: "25c104b644e00e4bab68ea5293f0d587"
server_key_version: "encryptionkey@0"
I20260925 02:53:02.757992 16761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:02.777017 16761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:02.779435 16761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:02.779619 16761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:02.785961 16761 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:38151
I20260925 02:53:02.785991 16828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:38151 every 8 connection(s)
I20260925 02:53:02.786757 16761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:02.787091 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 16761
I20260925 02:53:02.787233 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:53:02.787485 13179 external_mini_cluster.cc:1428] Setting key 001c5da664b588ef9cf7c2dbfa660d4a
I20260925 02:53:02.789549 16829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:02.794462 16829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c: Bootstrap starting.
Sep 25 02:53:02 dist-test-slave-252r krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:02.796833 16829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:02.797513 16829 log.cc:824] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:02.799607 16829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c: No bootstrap required, opened a new log
I20260925 02:53:02.800589 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:02.788738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58332 (local address 127.12.222.254:38151)
0925 02:53:02.789133 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:02.789146 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:02.789178 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:53:02.789801 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:02.789818 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:02.790108 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:02.790408 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:02.791173 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:02.792046 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:02.792886 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:02.793304 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:02.793520 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:02.795902 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:02.795907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:02.795918 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:02.795962 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:02.797646 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:02.798151 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:02.798155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:02.798159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:02.798315 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:02.798589 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:02.798592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:02.798594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:02.799049 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:02.799224 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:02.799444 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 02:53:02.799711 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":137,"threads_started":1}
I20260925 02:53:02.802947 16829 raft_consensus.cc:348] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef4eec7ff47429c92e568cdbe618d3c" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38151 } }
I20260925 02:53:02.803128 16829 raft_consensus.cc:374] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:02.803164 16829 raft_consensus.cc:729] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bef4eec7ff47429c92e568cdbe618d3c, State: Initialized, Role: FOLLOWER
I20260925 02:53:02.803498 16829 consensus_queue.cc:260] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef4eec7ff47429c92e568cdbe618d3c" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38151 } }
I20260925 02:53:02.803602 16829 raft_consensus.cc:388] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:02.803687 16829 raft_consensus.cc:482] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:02.803777 16829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:02.804632 16829 raft_consensus.cc:504] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef4eec7ff47429c92e568cdbe618d3c" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38151 } }
I20260925 02:53:02.804949 16829 leader_election.cc:302] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bef4eec7ff47429c92e568cdbe618d3c; no voters:
I20260925 02:53:02.805224 16829 leader_election.cc:288] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:53:02.805352 16834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:02.805599 16834 raft_consensus.cc:686] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c [term 1 LEADER]: Becoming Leader. State: Replica: bef4eec7ff47429c92e568cdbe618d3c, State: Running, Role: LEADER
I20260925 02:53:02.805907 16834 consensus_queue.cc:237] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef4eec7ff47429c92e568cdbe618d3c" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38151 } }
I20260925 02:53:02.806375 16829 sys_catalog.cc:564] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:02.807430 16835 sys_catalog.cc:455] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bef4eec7ff47429c92e568cdbe618d3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef4eec7ff47429c92e568cdbe618d3c" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38151 } } }
I20260925 02:53:02.807562 16835 sys_catalog.cc:458] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:02.807909 16843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:02.807888 16836 sys_catalog.cc:455] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bef4eec7ff47429c92e568cdbe618d3c. Latest consensus state: current_term: 1 leader_uuid: "bef4eec7ff47429c92e568cdbe618d3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef4eec7ff47429c92e568cdbe618d3c" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38151 } } }
I20260925 02:53:02.808593 16836 sys_catalog.cc:458] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:02.810782 16843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:02.815433 16843 catalog_manager.cc:1347] Generated new cluster ID: 3993cf665490442b9372dcbc74500b0f
I20260925 02:53:02.815500 16843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:02.832154 16843 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:02.832930 16843 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:02.845542 16843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bef4eec7ff47429c92e568cdbe618d3c: Generated new TSK 0
I20260925 02:53:02.846132 16843 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
I20260925 02:53:02.879258 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38151
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:02.977891 16856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:02.978142 16856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:02.978223 16856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:02.981772 16856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:02.981846 16856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:02.981927 16856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:53:02.986455 16856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:02.987258 16856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:02.988098 16856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:02.996474 16864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:02.996568 16856 server_base.cc:1034] running on GCE node
W20260925 02:53:02.996485 16862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:02.996754 16861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:02.997102 16856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:02.997663 16856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:02.998813 16856 hybrid_clock.cc:648] HybridClock initialized: now 1790304782998784 us; error 42 us; skew 500 ppm
Sep 25 02:53:02 dist-test-slave-252r krb5kdc[16748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304782, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:03.001498 16856 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:53:03.002202 16856 webserver.cc:460] Webserver started at http://127.12.222.193:41613/ using document root <none> and password file <none>
I20260925 02:53:03.002657 16856 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:03.002734 16856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:03.002949 16856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:03.004801 16856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "58f6826a4cef4bb8bd648136a357bd1f"
format_stamp: "Formatted at 2026-09-25 02:53:02 on dist-test-slave-252r"
server_key: "d9a4ab91621ce0f1d30a45f14ce8760c"
server_key_iv: "696c5d6a0ce87753be66501ec9e3e9c1"
server_key_version: "encryptionkey@0"
I20260925 02:53:03.005295 16856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "58f6826a4cef4bb8bd648136a357bd1f"
format_stamp: "Formatted at 2026-09-25 02:53:02 on dist-test-slave-252r"
server_key: "d9a4ab91621ce0f1d30a45f14ce8760c"
server_key_iv: "696c5d6a0ce87753be66501ec9e3e9c1"
server_key_version: "encryptionkey@0"
I20260925 02:53:03.009016 16856 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:53:03.011332 16871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:03.012442 16856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:53:03.012588 16856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "58f6826a4cef4bb8bd648136a357bd1f"
format_stamp: "Formatted at 2026-09-25 02:53:02 on dist-test-slave-252r"
server_key: "d9a4ab91621ce0f1d30a45f14ce8760c"
server_key_iv: "696c5d6a0ce87753be66501ec9e3e9c1"
server_key_version: "encryptionkey@0"
I20260925 02:53:03.012684 16856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:03.035755 16856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:03.038770 16856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:03.038949 16856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:03.039453 16856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:03.040149 16856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:03.040215 16856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:03.040293 16856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:03.040335 16856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:03.048655 16856 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:43627
I20260925 02:53:03.048679 16984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:43627 every 8 connection(s)
I20260925 02:53:03.049350 16856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:53:03.054250 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 16856
I20260925 02:53:03.054368 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:53:03.054664 13179 external_mini_cluster.cc:1428] Setting key f38e81bb4836cadbf9206fdb66c25c26
Sep 25 02:53:03 dist-test-slave-252r krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304782, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:03.062623 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:03.050865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:49665 (local address 127.12.222.254:38151)
0925 02:53:03.051033 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:03.051043 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:03.051737 (+ 694us) server_negotiation.cc:407] Connection header received
0925 02:53:03.053078 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:03.053081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:03.053134 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:03.053231 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:03.054487 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:03.055037 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:03.056330 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:03.056550 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:03.056657 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:03.059135 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:03.059138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:03.059140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:03.059161 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:03.060660 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:03.061179 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:03.061183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:03.061184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:03.061268 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:03.061597 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:03.061601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:03.061602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:03.061792 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:03.061901 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:03.062354 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 02:53:03.062476 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 02:53:03.063270 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:03.051171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38151 (local address 127.12.222.193:49665)
0925 02:53:03.051601 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:03.051640 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:03.052880 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:03.053406 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:03.053413 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:03.053728 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:03.054299 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:03.054320 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:03.055159 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:03.055164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:03.056148 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:03.056158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:03.056695 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:03.056698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:03.056850 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:03.057357 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:03.057371 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:03.058981 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:03.060762 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:03.060766 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:03.060778 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:03.061071 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:03.061360 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:03.061364 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:03.061366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:03.061500 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:03.061899 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:03.061904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:03.062138 (+ 234us) client_negotiation.cc:769] Sending connection context
0925 02:53:03.062315 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 02:53:03.062573 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.12.222.194@KRBTEST.COM; defaulting to no policy
I20260925 02:53:03.064530 16985 heartbeater.cc:347] Connected to a master server at 127.12.222.254:38151
I20260925 02:53:03.064747 16985 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:03.065158 16985 heartbeater.cc:511] Master 127.12.222.254:38151 requested a full tablet report, sending...
I20260925 02:53:03.066524 16793 ts_manager.cc:194] Registered new tserver with Master: 58f6826a4cef4bb8bd648136a357bd1f (127.12.222.193:43627)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.194@KRBTEST.COM" created.
I20260925 02:53:03.067780 16793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:49665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
I20260925 02:53:03.083006 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38151
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:03.180503 16990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:03.180704 16990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:03.180738 16990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:03.185070 16990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:03.185117 16990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:03.185212 16990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:53:03.189929 16990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:03.190793 16990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:03.191732 16990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:03.200316 16996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:03.200284 16998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:03.200287 16995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:03.200584 16990 server_base.cc:1034] running on GCE node
I20260925 02:53:03.200989 16990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:03.201548 16990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:03.202723 16990 hybrid_clock.cc:648] HybridClock initialized: now 1790304783202713 us; error 24 us; skew 500 ppm
Sep 25 02:53:03 dist-test-slave-252r krb5kdc[16748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304783, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:03.205293 16990 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:53:03.206095 16990 webserver.cc:460] Webserver started at http://127.12.222.194:38769/ using document root <none> and password file <none>
I20260925 02:53:03.206573 16990 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:03.206652 16990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:03.206857 16990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:03.208847 16990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "434bd45487a940259702504753ad2d3d"
format_stamp: "Formatted at 2026-09-25 02:53:03 on dist-test-slave-252r"
server_key: "335e1d136de47043b0c9bbb5e82a9d5a"
server_key_iv: "b1938cbf94ef975cfae2e8b4570236e8"
server_key_version: "encryptionkey@0"
I20260925 02:53:03.209419 16990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "434bd45487a940259702504753ad2d3d"
format_stamp: "Formatted at 2026-09-25 02:53:03 on dist-test-slave-252r"
server_key: "335e1d136de47043b0c9bbb5e82a9d5a"
server_key_iv: "b1938cbf94ef975cfae2e8b4570236e8"
server_key_version: "encryptionkey@0"
I20260925 02:53:03.212844 16990 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:53:03.215065 17005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:03.216356 16990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:53:03.216470 16990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "434bd45487a940259702504753ad2d3d"
format_stamp: "Formatted at 2026-09-25 02:53:03 on dist-test-slave-252r"
server_key: "335e1d136de47043b0c9bbb5e82a9d5a"
server_key_iv: "b1938cbf94ef975cfae2e8b4570236e8"
server_key_version: "encryptionkey@0"
I20260925 02:53:03.216583 16990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:03.235100 16990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:03.237881 16990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:03.238065 16990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:03.238554 16990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:03.239246 16990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:03.239310 16990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:03.239373 16990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:03.239413 16990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:03.248726 16990 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:42165
I20260925 02:53:03.248750 17119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:42165 every 8 connection(s)
I20260925 02:53:03.249471 16990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:53:03 dist-test-slave-252r krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304783, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:03.258796 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 16990
I20260925 02:53:03.258936 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:53:03.259187 13179 external_mini_cluster.cc:1428] Setting key 1974373947ce5a699ae3919fc200b770
I20260925 02:53:03.261133 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:03.251021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:60951 (local address 127.12.222.254:38151)
0925 02:53:03.251147 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:03.251150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:03.251851 (+ 701us) server_negotiation.cc:407] Connection header received
0925 02:53:03.253090 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:03.253093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:03.253137 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:03.253205 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:03.254210 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:03.254708 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:03.255661 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:03.255853 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:03.255920 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:03.257947 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:03.257950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:03.257951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:03.257971 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:03.259207 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:03.259683 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:03.259687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:03.259688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:03.259764 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:03.260079 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:03.260082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:03.260084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:03.260289 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:03.260399 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:03.260830 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 02:53:03.260940 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:53:03.261823 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:03.251323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38151 (local address 127.12.222.194:60951)
0925 02:53:03.251714 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:03.251750 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:03.252910 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:03.253346 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:03.253352 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:03.253683 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:03.254093 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:03.254105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:03.254803 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:03.254805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:03.255567 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:03.255575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:03.255966 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:03.255968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:03.256108 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:03.256584 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:03.256598 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:03.257806 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:03.259336 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:03.259341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:03.259352 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:03.259586 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:03.259868 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:03.259870 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:03.259872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:03.259991 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:03.260399 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:03.260405 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:03.260612 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 02:53:03.260782 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 02:53:03.261095 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
I20260925 02:53:03.263012 17120 heartbeater.cc:347] Connected to a master server at 127.12.222.254:38151
I20260925 02:53:03.263234 17120 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:03.263788 17120 heartbeater.cc:511] Master 127.12.222.254:38151 requested a full tablet report, sending...
I20260925 02:53:03.264747 16793 ts_manager.cc:194] Registered new tserver with Master: 434bd45487a940259702504753ad2d3d (127.12.222.194:42165)
I20260925 02:53:03.265254 16793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:60951
WARNING: no policy specified for kudu/127.12.222.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
I20260925 02:53:03.285231 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38151
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:03.386123 17125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:03.386358 17125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:03.386431 17125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:03.389991 17125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:03.390061 17125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:03.390141 17125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:53:03.394807 17125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:03.395681 17125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:03.396592 17125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:03.405066 17131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:03.405063 17130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:03.405117 17133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:03.405225 17125 server_base.cc:1034] running on GCE node
I20260925 02:53:03.405647 17125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:03.406153 17125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:03.407300 17125 hybrid_clock.cc:648] HybridClock initialized: now 1790304783407277 us; error 35 us; skew 500 ppm
Sep 25 02:53:03 dist-test-slave-252r krb5kdc[16748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304783, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:03.409803 17125 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:53:03.410580 17125 webserver.cc:460] Webserver started at http://127.12.222.195:37247/ using document root <none> and password file <none>
I20260925 02:53:03.411073 17125 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:03.411127 17125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:03.411356 17125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:03.413146 17125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "8391aab57ba3469dbb340d3a39e14026"
format_stamp: "Formatted at 2026-09-25 02:53:03 on dist-test-slave-252r"
server_key: "85b60ad503b38417a27b398ce3001a56"
server_key_iv: "a18dbab26cef3a4851681db18f031880"
server_key_version: "encryptionkey@0"
I20260925 02:53:03.413677 17125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "8391aab57ba3469dbb340d3a39e14026"
format_stamp: "Formatted at 2026-09-25 02:53:03 on dist-test-slave-252r"
server_key: "85b60ad503b38417a27b398ce3001a56"
server_key_iv: "a18dbab26cef3a4851681db18f031880"
server_key_version: "encryptionkey@0"
I20260925 02:53:03.417022 17125 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:53:03.419020 17140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:03.420003 17125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:53:03.420104 17125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "8391aab57ba3469dbb340d3a39e14026"
format_stamp: "Formatted at 2026-09-25 02:53:03 on dist-test-slave-252r"
server_key: "85b60ad503b38417a27b398ce3001a56"
server_key_iv: "a18dbab26cef3a4851681db18f031880"
server_key_version: "encryptionkey@0"
I20260925 02:53:03.420189 17125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:03.448975 17125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:03.451299 17125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:03.451502 17125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:03.452044 17125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:03.452765 17125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:03.452808 17125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:03.452868 17125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:03.452899 17125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:03.460791 17125 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:42445
I20260925 02:53:03.460837 17255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:42445 every 8 connection(s)
I20260925 02:53:03.461645 17125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:53:03 dist-test-slave-252r krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304783, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:03.470824 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 17125
I20260925 02:53:03.470957 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:53:03.471280 13179 external_mini_cluster.cc:1428] Setting key af9c20ff2999ae3d885113a6c92a307c
I20260925 02:53:03.473773 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:03.463138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:42349 (local address 127.12.222.254:38151)
0925 02:53:03.463264 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:03.463267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:03.463946 (+ 679us) server_negotiation.cc:407] Connection header received
0925 02:53:03.465238 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:03.465242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:03.465300 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:03.465376 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:03.466382 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:03.466874 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:03.467899 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:03.468137 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:03.468239 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:03.470285 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:03.470289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:03.470291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:03.470320 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:03.471692 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:03.472324 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:03.472327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:03.472328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:03.472405 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:03.472765 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:03.472767 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:03.472768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:03.472951 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:03.473019 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:03.473574 (+ 555us) server_negotiation.cc:299] Negotiation successful
0925 02:53:03.473667 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:53:03.474521 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:03.463398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38151 (local address 127.12.222.195:42349)
0925 02:53:03.463800 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:03.463837 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:03.465037 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:03.465531 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:03.465537 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:03.465831 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:03.466235 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:03.466247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:03.467004 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:03.467012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:03.467774 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:03.467782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:03.468305 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:03.468309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:03.468456 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:03.468875 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:03.468889 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:03.470128 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:03.471808 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:03.471814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:03.471826 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:03.472173 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:03.472529 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:03.472531 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:03.472532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:03.472657 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:03.473045 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:03.473051 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:03.473300 (+ 249us) client_negotiation.cc:769] Sending connection context
0925 02:53:03.473511 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 02:53:03.473751 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":101,"threads_started":1}
I20260925 02:53:03.475440 17256 heartbeater.cc:347] Connected to a master server at 127.12.222.254:38151
I20260925 02:53:03.475648 17256 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:03.476091 17256 heartbeater.cc:511] Master 127.12.222.254:38151 requested a full tablet report, sending...
I20260925 02:53:03.476999 16793 ts_manager.cc:194] Registered new tserver with Master: 8391aab57ba3469dbb340d3a39e14026 (127.12.222.195:42445)
I20260925 02:53:03.477469 16793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:42349
I20260925 02:53:03.485031 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:53:03 dist-test-slave-252r krb5kdc[16748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:53:03 dist-test-slave-252r krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.193@KRBTEST.COM
I20260925 02:53:03.507632 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:03.499425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48726 (local address 127.12.222.193:43627)
0925 02:53:03.499731 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:03.499739 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:03.499760 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:53:03.499800 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:03.499803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:03.499960 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:03.500062 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:03.500696 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:03.501454 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:03.502187 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:03.502508 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:03.502582 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:03.504676 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:03.504684 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:03.504694 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:03.504714 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:03.505987 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:03.506363 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:03.506367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:03.506368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:03.506440 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:03.506659 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:03.506661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:03.506662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:03.506969 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:03.507134 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:03.507371 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 02:53:03.507488 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":109,"threads_started":1}
Sep 25 02:53:03 dist-test-slave-252r krb5kdc[16748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:03.517424 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:03.510261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58334 (local address 127.12.222.254:38151)
0925 02:53:03.510421 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:03.510424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:03.510462 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:53:03.510566 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:03.510569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:03.510604 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:03.510668 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:03.511263 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:03.511882 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:03.512576 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:03.512757 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:03.512824 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:03.514866 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:03.514869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:03.514870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:03.514889 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:03.516044 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:03.516430 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:03.516432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:03.516433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:03.516503 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:03.516699 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:03.516701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:03.516701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:03.516853 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:03.516936 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:03.517169 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 02:53:03.517269 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 02:53:03.517768 16793 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58334
I20260925 02:53:03.518302 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 16856
I20260925 02:53:03.525137 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 16990
I20260925 02:53:03.530314 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 17125
I20260925 02:53:03.535480 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 16761
2026-09-25T02:53:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2010 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:53:03 dist-test-slave-252r krb5kdc[17268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:53:03 dist-test-slave-252r krb5kdc[17268](info): set up 2 sockets
Sep 25 02:53:03 dist-test-slave-252r krb5kdc[17268](info): commencing operation
krb5kdc: starting...
W20260925 02:53:04.520216 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:53:04 dist-test-slave-252r krb5kdc[17268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:53:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:53:04.631345 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:45313
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:45313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:04.735673 17281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:04.735919 17281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:04.735965 17281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:04.739796 17281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:04.739848 17281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:04.739866 17281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:04.739881 17281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:04.739897 17281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:04.744866 17281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:45313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:45313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:04.745749 17281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:04.746670 17281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:04.754379 17287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:04.754367 17286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:04.754387 17289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:04.754675 17281 server_base.cc:1034] running on GCE node
I20260925 02:53:04.755195 17281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:04.756253 17281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:04.757422 17281 hybrid_clock.cc:648] HybridClock initialized: now 1790304784757405 us; error 28 us; skew 500 ppm
Sep 25 02:53:04 dist-test-slave-252r krb5kdc[17268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304784, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:04.763284 17281 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:53:04.763960 17281 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:04.764021 17281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:04.764261 17281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:04.766137 17281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "e5355a596c6940bd9ee6e35a11cfe44d"
format_stamp: "Formatted at 2026-09-25 02:53:04 on dist-test-slave-252r"
server_key: "cf9e7bad958a8d23cad96a9d8e00ada5"
server_key_iv: "582bcec931a1cd73f889da70cac18271"
server_key_version: "encryptionkey@0"
I20260925 02:53:04.766680 17281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "e5355a596c6940bd9ee6e35a11cfe44d"
format_stamp: "Formatted at 2026-09-25 02:53:04 on dist-test-slave-252r"
server_key: "cf9e7bad958a8d23cad96a9d8e00ada5"
server_key_iv: "582bcec931a1cd73f889da70cac18271"
server_key_version: "encryptionkey@0"
I20260925 02:53:04.770215 17281 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:53:04.772423 17295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:04.773597 17281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:53:04.773737 17281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "e5355a596c6940bd9ee6e35a11cfe44d"
format_stamp: "Formatted at 2026-09-25 02:53:04 on dist-test-slave-252r"
server_key: "cf9e7bad958a8d23cad96a9d8e00ada5"
server_key_iv: "582bcec931a1cd73f889da70cac18271"
server_key_version: "encryptionkey@0"
I20260925 02:53:04.773844 17281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:04.787266 17281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:04.789889 17281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:04.790079 17281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:04.796962 17281 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:45313
I20260925 02:53:04.797233 17348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:45313 every 8 connection(s)
I20260925 02:53:04.797592 17281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:04.800177 17349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:04.805394 17349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d: Bootstrap starting.
I20260925 02:53:04.806723 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 17281
I20260925 02:53:04.806849 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:53:04.807117 13179 external_mini_cluster.cc:1428] Setting key e5b45187bfa0a709e0f340b7a42a878f
I20260925 02:53:04.807953 17349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:04.808775 17349 log.cc:824] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:04.810782 17349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d: No bootstrap required, opened a new log
I20260925 02:53:04.814918 17349 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5355a596c6940bd9ee6e35a11cfe44d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 45313 } }
I20260925 02:53:04.815217 17349 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:04.815281 17349 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5355a596c6940bd9ee6e35a11cfe44d, State: Initialized, Role: FOLLOWER
I20260925 02:53:04.815764 17349 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5355a596c6940bd9ee6e35a11cfe44d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 45313 } }
I20260925 02:53:04.815922 17349 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 25 02:53:04 dist-test-slave-252r krb5kdc[17268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:04.815995 17349 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:04.816094 17349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:04.817072 17349 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5355a596c6940bd9ee6e35a11cfe44d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 45313 } }
I20260925 02:53:04.817397 17349 leader_election.cc:302] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5355a596c6940bd9ee6e35a11cfe44d; no voters:
I20260925 02:53:04.817699 17349 leader_election.cc:288] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:53:04.817794 17354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:04.818004 17354 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d [term 1 LEADER]: Becoming Leader. State: Replica: e5355a596c6940bd9ee6e35a11cfe44d, State: Running, Role: LEADER
I20260925 02:53:04.818284 17354 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5355a596c6940bd9ee6e35a11cfe44d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 45313 } }
I20260925 02:53:04.818856 17349 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:04.819671 17356 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5355a596c6940bd9ee6e35a11cfe44d. Latest consensus state: current_term: 1 leader_uuid: "e5355a596c6940bd9ee6e35a11cfe44d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5355a596c6940bd9ee6e35a11cfe44d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 45313 } } }
I20260925 02:53:04.819788 17356 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:04.820431 17355 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5355a596c6940bd9ee6e35a11cfe44d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5355a596c6940bd9ee6e35a11cfe44d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 45313 } } }
I20260925 02:53:04.820526 17355 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:04.822098 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:04.808207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56162 (local address 127.12.222.254:45313)
0925 02:53:04.808669 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:04.808687 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:04.808720 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:53:04.809489 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:04.809514 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:04.809804 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:04.810082 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:04.810938 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:04.811864 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:04.813124 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:04.813550 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:04.813673 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:04.816692 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:04.816697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:04.816708 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:04.816732 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:04.818626 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:04.819145 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:04.819150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:04.819155 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:04.819298 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:04.819599 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:04.819602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:04.819604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:04.820102 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:04.820318 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:04.821031 (+ 713us) server_negotiation.cc:299] Negotiation successful
0925 02:53:04.821315 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":113,"threads_started":1}
W20260925 02:53:04.823230 17369 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:53:04.823325 17369 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:53:04.823422 17363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:04.824268 17363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:04.829730 17363 catalog_manager.cc:1347] Generated new cluster ID: 203e3e4d6fde482d8279c63438be7655
I20260925 02:53:04.829798 17363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:04.852974 17363 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:04.853860 17363 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:04.857748 17363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d: Generated new TSK 0
I20260925 02:53:04.858410 17363 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
I20260925 02:53:04.892922 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:45313
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:04.993691 17375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:04.993995 17375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:04.994071 17375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:04.998360 17375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:04.998407 17375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:04.998457 17375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:53:05.003180 17375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:45313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:05.003989 17375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:05.004885 17375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:05.013506 17380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:05.013525 17381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:05.013585 17375 server_base.cc:1034] running on GCE node
W20260925 02:53:05.013525 17383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:05.014074 17375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:05.014616 17375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:05.015786 17375 hybrid_clock.cc:648] HybridClock initialized: now 1790304785015769 us; error 26 us; skew 500 ppm
Sep 25 02:53:05 dist-test-slave-252r krb5kdc[17268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304785, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:05.018186 17375 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:53:05.018872 17375 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:05.018954 17375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:05.019145 17375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:05.021184 17375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "bd60d473ccb345cdb911cb6da271bdac"
format_stamp: "Formatted at 2026-09-25 02:53:05 on dist-test-slave-252r"
server_key: "b91db31b6e4ff6912168ebd181e04e1f"
server_key_iv: "fe5c62cbad2420bd906e5c76489a32bd"
server_key_version: "encryptionkey@0"
I20260925 02:53:05.021704 17375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "bd60d473ccb345cdb911cb6da271bdac"
format_stamp: "Formatted at 2026-09-25 02:53:05 on dist-test-slave-252r"
server_key: "b91db31b6e4ff6912168ebd181e04e1f"
server_key_iv: "fe5c62cbad2420bd906e5c76489a32bd"
server_key_version: "encryptionkey@0"
I20260925 02:53:05.025105 17375 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:53:05.027221 17389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:05.028386 17375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:53:05.028518 17375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "bd60d473ccb345cdb911cb6da271bdac"
format_stamp: "Formatted at 2026-09-25 02:53:05 on dist-test-slave-252r"
server_key: "b91db31b6e4ff6912168ebd181e04e1f"
server_key_iv: "fe5c62cbad2420bd906e5c76489a32bd"
server_key_version: "encryptionkey@0"
I20260925 02:53:05.028615 17375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:05.049062 17375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:05.051422 17375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:05.051597 17375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:05.052110 17375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:05.052862 17375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:05.052950 17375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:05.053013 17375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:05.053056 17375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:05.061198 17375 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:35817
I20260925 02:53:05.061230 17503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:35817 every 8 connection(s)
I20260925 02:53:05.061717 17375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:53:05.068001 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 17375
I20260925 02:53:05.068130 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:53:05.068421 13179 external_mini_cluster.cc:1428] Setting key 933799314465dcbb0b42c1fbabca6435
Sep 25 02:53:05 dist-test-slave-252r krb5kdc[17268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304785, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:05.074002 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.063178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:48837 (local address 127.12.222.254:45313)
0925 02:53:05.063304 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:05.063307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:05.063988 (+ 681us) server_negotiation.cc:407] Connection header received
0925 02:53:05.065258 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:05.065261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:05.065322 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:05.065424 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:05.066462 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.066923 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.067956 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.068256 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.068348 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:05.070570 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:05.070573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:05.070575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:05.070603 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:05.071952 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:05.072529 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:05.072531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:05.072533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:05.072602 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:05.072963 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:05.072965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:05.072966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:05.073106 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:05.073173 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:05.073782 (+ 609us) server_negotiation.cc:299] Negotiation successful
0925 02:53:05.073884 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:53:05.075078 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.063438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:45313 (local address 127.12.222.193:48837)
0925 02:53:05.063843 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:05.063881 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:05.065042 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:05.065595 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:05.065602 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:05.065923 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:05.066318 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:05.066329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.067042 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.067047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:05.067827 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:05.067835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.068382 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.068385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:05.068577 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:05.069121 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:05.069135 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:05.070422 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:05.072063 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:05.072071 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:05.072086 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:05.072441 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:05.072697 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:05.072702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:05.072707 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:05.072875 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:05.073205 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:05.073214 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:05.073489 (+ 275us) client_negotiation.cc:769] Sending connection context
0925 02:53:05.073722 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 02:53:05.074053 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":111,"threads_started":1}
I20260925 02:53:05.076215 17504 heartbeater.cc:347] Connected to a master server at 127.12.222.254:45313
I20260925 02:53:05.076478 17504 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:05.076906 17504 heartbeater.cc:511] Master 127.12.222.254:45313 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.222.194@KRBTEST.COM; defaulting to no policy
I20260925 02:53:05.078316 17312 ts_manager.cc:194] Registered new tserver with Master: bd60d473ccb345cdb911cb6da271bdac (127.12.222.193:35817)
I20260925 02:53:05.079494 17312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:48837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
I20260925 02:53:05.093211 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:45313
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:05.191308 17509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:05.191547 17509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:05.191622 17509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:05.195274 17509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:05.195346 17509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:05.195431 17509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:53:05.200218 17509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:45313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:05.201191 17509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:05.202111 17509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:05.210592 17517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:05.210588 17514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:05.210587 17515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:05.210842 17509 server_base.cc:1034] running on GCE node
I20260925 02:53:05.211210 17509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:05.211761 17509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:05.212932 17509 hybrid_clock.cc:648] HybridClock initialized: now 1790304785212920 us; error 28 us; skew 500 ppm
Sep 25 02:53:05 dist-test-slave-252r krb5kdc[17268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304785, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:05.215539 17509 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:53:05.216141 17509 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:05.216223 17509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:05.216454 17509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:05.218158 17509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "4a896c0ddd8b4d6386e1227d9308ad90"
format_stamp: "Formatted at 2026-09-25 02:53:05 on dist-test-slave-252r"
server_key: "f23e2e7592e6cffcc64743b22546553c"
server_key_iv: "8f51a4bbd7d097e980e9fe5ae3f5d12f"
server_key_version: "encryptionkey@0"
I20260925 02:53:05.218767 17509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "4a896c0ddd8b4d6386e1227d9308ad90"
format_stamp: "Formatted at 2026-09-25 02:53:05 on dist-test-slave-252r"
server_key: "f23e2e7592e6cffcc64743b22546553c"
server_key_iv: "8f51a4bbd7d097e980e9fe5ae3f5d12f"
server_key_version: "encryptionkey@0"
I20260925 02:53:05.222462 17509 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:53:05.224560 17523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:05.225651 17509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:05.225773 17509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "4a896c0ddd8b4d6386e1227d9308ad90"
format_stamp: "Formatted at 2026-09-25 02:53:05 on dist-test-slave-252r"
server_key: "f23e2e7592e6cffcc64743b22546553c"
server_key_iv: "8f51a4bbd7d097e980e9fe5ae3f5d12f"
server_key_version: "encryptionkey@0"
I20260925 02:53:05.225873 17509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:05.239183 17509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:05.241757 17509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:05.241942 17509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:05.242486 17509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:05.243238 17509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:05.243315 17509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:05.243381 17509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:05.243432 17509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:05.251472 17509 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:32885
I20260925 02:53:05.251529 17637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:32885 every 8 connection(s)
I20260925 02:53:05.252017 17509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:53:05.258074 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 17509
I20260925 02:53:05.258172 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:53:05.258440 13179 external_mini_cluster.cc:1428] Setting key d814045fb8cce5d6ec6d69980f6c7f16
Sep 25 02:53:05 dist-test-slave-252r krb5kdc[17268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304785, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:05.264894 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.253563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:52525 (local address 127.12.222.254:45313)
0925 02:53:05.253751 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:05.253756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:05.254463 (+ 707us) server_negotiation.cc:407] Connection header received
0925 02:53:05.255598 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:05.255601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:05.255645 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:05.255711 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:05.256756 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.257265 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.258302 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.258558 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.258669 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:05.261120 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:05.261123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:05.261138 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:05.261168 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:05.262773 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:05.263211 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:05.263217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:05.263221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:05.263309 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:05.263632 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:05.263638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:05.263643 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:05.263899 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:05.264006 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:05.264507 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 02:53:05.264642 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 02:53:05.265401 17640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.253923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:45313 (local address 127.12.222.194:52525)
0925 02:53:05.254319 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:05.254358 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:05.255407 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:05.255851 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:05.255857 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:05.256181 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:05.256641 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:05.256654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.257361 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.257363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:05.258183 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:05.258195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.258713 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.258717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:05.258863 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:05.259475 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:05.259500 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:05.260965 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:05.262875 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:05.262880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:05.262889 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:05.263113 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:05.263398 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:05.263400 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:05.263401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:05.263518 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:05.264017 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:05.264022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:05.264216 (+ 194us) client_negotiation.cc:769] Sending connection context
0925 02:53:05.264421 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 02:53:05.264648 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":103,"threads_started":1}
I20260925 02:53:05.266425 17638 heartbeater.cc:347] Connected to a master server at 127.12.222.254:45313
I20260925 02:53:05.266649 17638 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:05.267045 17638 heartbeater.cc:511] Master 127.12.222.254:45313 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.222.195@KRBTEST.COM; defaulting to no policy
I20260925 02:53:05.268005 17312 ts_manager.cc:194] Registered new tserver with Master: 4a896c0ddd8b4d6386e1227d9308ad90 (127.12.222.194:32885)
I20260925 02:53:05.268576 17312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:52525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
I20260925 02:53:05.283052 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:45313
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:05.383723 17643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:05.383978 17643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:05.384055 17643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:05.387686 17643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:05.387759 17643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:05.387844 17643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:53:05.392656 17643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:45313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:05.393512 17643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:05.394454 17643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:05.403057 17648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:05.403221 17643 server_base.cc:1034] running on GCE node
W20260925 02:53:05.403064 17649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:05.403061 17651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:05.403674 17643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:05.404219 17643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:05.405438 17643 hybrid_clock.cc:648] HybridClock initialized: now 1790304785405432 us; error 27 us; skew 500 ppm
Sep 25 02:53:05 dist-test-slave-252r krb5kdc[17268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304785, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:05.408020 17643 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:53:05.408669 17643 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:05.408757 17643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:05.408960 17643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:05.410876 17643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "b981e6f955d743158fdcd29082ec8f44"
format_stamp: "Formatted at 2026-09-25 02:53:05 on dist-test-slave-252r"
server_key: "8b3ba38802e73eac632ffded2b259a23"
server_key_iv: "06d4c94710d4a90fcd48def1284063c2"
server_key_version: "encryptionkey@0"
I20260925 02:53:05.411392 17643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "b981e6f955d743158fdcd29082ec8f44"
format_stamp: "Formatted at 2026-09-25 02:53:05 on dist-test-slave-252r"
server_key: "8b3ba38802e73eac632ffded2b259a23"
server_key_iv: "06d4c94710d4a90fcd48def1284063c2"
server_key_version: "encryptionkey@0"
I20260925 02:53:05.414673 17643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 02:53:05.416863 17657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:05.417917 17643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:05.418038 17643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "b981e6f955d743158fdcd29082ec8f44"
format_stamp: "Formatted at 2026-09-25 02:53:05 on dist-test-slave-252r"
server_key: "8b3ba38802e73eac632ffded2b259a23"
server_key_iv: "06d4c94710d4a90fcd48def1284063c2"
server_key_version: "encryptionkey@0"
I20260925 02:53:05.418134 17643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:05.425979 17643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:05.428339 17643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:05.428498 17643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:05.428963 17643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:05.429689 17643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:05.429751 17643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:05.429803 17643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:05.429845 17643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:05.437897 17643 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:38889
I20260925 02:53:05.437920 17770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:38889 every 8 connection(s)
I20260925 02:53:05.438354 17643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:53:05 dist-test-slave-252r krb5kdc[17268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304785, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:05.447989 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 17643
I20260925 02:53:05.448089 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:53:05.448421 13179 external_mini_cluster.cc:1428] Setting key a11189a228cd14864905d7c7010fb009
I20260925 02:53:05.450024 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.439740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:34615 (local address 127.12.222.254:45313)
0925 02:53:05.439852 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:05.439856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:05.440554 (+ 698us) server_negotiation.cc:407] Connection header received
0925 02:53:05.441694 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:05.441697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:05.441765 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:05.441835 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:05.442788 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.443306 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.444490 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.444728 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.444835 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:05.447001 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:05.447004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:05.447006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:05.447028 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:05.448136 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:05.448658 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:05.448661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:05.448663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:05.448740 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:05.449062 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:05.449065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:05.449066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:05.449230 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:05.449328 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:05.449758 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 02:53:05.449862 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:53:05.450641 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.440015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:45313 (local address 127.12.222.195:34615)
0925 02:53:05.440411 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:05.440451 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:05.441486 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:05.441979 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:05.441985 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:05.442272 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:05.442654 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:05.442665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.443450 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.443453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:05.444365 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:05.444374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.444866 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.444870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:05.445029 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:05.445511 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:05.445526 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:05.446857 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:05.448280 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:05.448285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:05.448296 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:05.448558 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:05.448833 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:05.448836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:05.448837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:05.448962 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:05.449329 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:05.449336 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:05.449543 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 02:53:05.449701 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 02:53:05.449959 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":92,"threads_started":1}
I20260925 02:53:05.451547 17771 heartbeater.cc:347] Connected to a master server at 127.12.222.254:45313
I20260925 02:53:05.451740 17771 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:05.452131 17771 heartbeater.cc:511] Master 127.12.222.254:45313 requested a full tablet report, sending...
I20260925 02:53:05.453037 17312 ts_manager.cc:194] Registered new tserver with Master: b981e6f955d743158fdcd29082ec8f44 (127.12.222.195:38889)
I20260925 02:53:05.453559 17312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:34615
I20260925 02:53:05.461844 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:53:05.469687 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.463520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56174 (local address 127.12.222.254:45313)
0925 02:53:05.463629 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:05.463632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:05.463868 (+ 236us) server_negotiation.cc:407] Connection header received
0925 02:53:05.463964 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:05.463966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:05.463999 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:05.464061 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:05.464761 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.465238 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.465981 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.466171 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.466236 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:05.467359 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:05.467362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:05.467363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:05.467383 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:05.468501 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:05.468839 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:05.468841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:05.468842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:05.468905 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:05.469113 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:05.469115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:05.469116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:05.469285 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:05.469377 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:05.469516 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 02:53:05.469591 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:53:05.472095 17312 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:53:05.474977 17312 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:53:05.487910 17783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.482415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:38889 (local address 127.0.0.1:38716)
0925 02:53:05.482967 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:05.483435 (+ 468us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:05.483523 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:05.484213 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:05.484215 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:05.484275 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:05.484361 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:05.484383 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.485327 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.485332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:05.486494 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:05.486503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.487398 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.487401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:05.487499 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:05.487530 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 02:53:05.487648 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 02:53:05.487766 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":51,"thread_start_us":75,"threads_started":1}
I20260925 02:53:05.487911 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.482920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.12.222.195:38889)
0925 02:53:05.483319 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:05.483333 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:05.483689 (+ 356us) server_negotiation.cc:407] Connection header received
0925 02:53:05.483767 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:05.483772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:05.483941 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:05.484101 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:05.484501 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.485212 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.486607 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.487259 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.487383 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:05.487491 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:05.487676 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 02:53:05.487774 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":144,"threads_started":1}
I20260925 02:53:05.488220 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.482916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:35817 (local address 127.0.0.1:41068)
0925 02:53:05.483708 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:05.483719 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:05.483785 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:05.484419 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:05.484424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:05.484453 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:05.484536 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:05.484544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.485766 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.485770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:05.486791 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:05.486799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.487894 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.487897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:05.487965 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:05.487990 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:53:05.488060 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 02:53:05.488118 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":25,"thread_start_us":94,"threads_started":1}
I20260925 02:53:05.488482 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.482394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51864 (local address 127.12.222.194:32885)
0925 02:53:05.482735 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:05.482747 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:05.483164 (+ 417us) server_negotiation.cc:407] Connection header received
0925 02:53:05.483245 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:05.483250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:05.483462 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:05.483605 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:05.484070 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.485215 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.486728 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.487276 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.487412 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:05.487519 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:05.488124 (+ 605us) server_negotiation.cc:299] Negotiation successful
0925 02:53:05.488199 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":130,"threads_started":1}
I20260925 02:53:05.488637 17782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.482245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:32885 (local address 127.0.0.1:51864)
0925 02:53:05.482869 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:05.482915 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:05.483031 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:05.483806 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:05.483812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:05.483833 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:05.483941 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:05.483947 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.485349 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.485352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:05.486604 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:05.486613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.487568 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.487573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:05.487667 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:05.487690 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:53:05.487763 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:53:05.487826 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":53,"thread_start_us":61,"threads_started":1}
I20260925 02:53:05.488783 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.483320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41068 (local address 127.12.222.193:35817)
0925 02:53:05.483968 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:05.483976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:05.483993 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:53:05.484085 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:05.484090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:05.484286 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:05.484439 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:05.484653 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.485616 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.486899 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.487606 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.487704 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:05.487829 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:05.488492 (+ 663us) server_negotiation.cc:299] Negotiation successful
0925 02:53:05.488641 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":116,"threads_started":1}
I20260925 02:53:05.490615 17705 tablet_service.cc:1467] Processing CreateTablet for tablet 4996bb9b6ae24f9c90ed5a04350585de (DEFAULT_TABLE table=test-table [id=ea60904927ef45e38d6d38463b3b11c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:05.491478 17705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4996bb9b6ae24f9c90ed5a04350585de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:05.491168 17571 tablet_service.cc:1467] Processing CreateTablet for tablet 4996bb9b6ae24f9c90ed5a04350585de (DEFAULT_TABLE table=test-table [id=ea60904927ef45e38d6d38463b3b11c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:05.491166 17437 tablet_service.cc:1467] Processing CreateTablet for tablet 4996bb9b6ae24f9c90ed5a04350585de (DEFAULT_TABLE table=test-table [id=ea60904927ef45e38d6d38463b3b11c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:05.492098 17571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4996bb9b6ae24f9c90ed5a04350585de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:05.492098 17437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4996bb9b6ae24f9c90ed5a04350585de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:05.498070 17788 tablet_bootstrap.cc:492] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44: Bootstrap starting.
I20260925 02:53:05.498692 17790 tablet_bootstrap.cc:492] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90: Bootstrap starting.
I20260925 02:53:05.499532 17789 tablet_bootstrap.cc:492] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac: Bootstrap starting.
I20260925 02:53:05.500463 17788 tablet_bootstrap.cc:654] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:05.500679 17790 tablet_bootstrap.cc:654] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:05.501546 17789 tablet_bootstrap.cc:654] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:05.501797 17790 log.cc:824] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:05.501797 17788 log.cc:824] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:05.502322 17789 log.cc:824] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:05.503710 17788 tablet_bootstrap.cc:492] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44: No bootstrap required, opened a new log
I20260925 02:53:05.503942 17788 ts_tablet_manager.cc:1397] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 02:53:05.504142 17789 tablet_bootstrap.cc:492] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac: No bootstrap required, opened a new log
I20260925 02:53:05.504182 17790 tablet_bootstrap.cc:492] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90: No bootstrap required, opened a new log
I20260925 02:53:05.504364 17789 ts_tablet_manager.cc:1397] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260925 02:53:05.504374 17790 ts_tablet_manager.cc:1397] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 02:53:05.506747 17789 raft_consensus.cc:348] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a896c0ddd8b4d6386e1227d9308ad90" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 32885 } } peers { permanent_uuid: "b981e6f955d743158fdcd29082ec8f44" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38889 } } peers { permanent_uuid: "bd60d473ccb345cdb911cb6da271bdac" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 35817 } }
I20260925 02:53:05.506973 17789 raft_consensus.cc:374] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:05.507021 17789 raft_consensus.cc:729] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd60d473ccb345cdb911cb6da271bdac, State: Initialized, Role: FOLLOWER
I20260925 02:53:05.507447 17789 consensus_queue.cc:260] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a896c0ddd8b4d6386e1227d9308ad90" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 32885 } } peers { permanent_uuid: "b981e6f955d743158fdcd29082ec8f44" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38889 } } peers { permanent_uuid: "bd60d473ccb345cdb911cb6da271bdac" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 35817 } }
I20260925 02:53:05.507488 17788 raft_consensus.cc:348] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a896c0ddd8b4d6386e1227d9308ad90" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 32885 } } peers { permanent_uuid: "b981e6f955d743158fdcd29082ec8f44" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38889 } } peers { permanent_uuid: "bd60d473ccb345cdb911cb6da271bdac" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 35817 } }
I20260925 02:53:05.507740 17788 raft_consensus.cc:374] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:05.507805 17788 raft_consensus.cc:729] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b981e6f955d743158fdcd29082ec8f44, State: Initialized, Role: FOLLOWER
I20260925 02:53:05.507753 17790 raft_consensus.cc:348] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a896c0ddd8b4d6386e1227d9308ad90" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 32885 } } peers { permanent_uuid: "b981e6f955d743158fdcd29082ec8f44" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38889 } } peers { permanent_uuid: "bd60d473ccb345cdb911cb6da271bdac" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 35817 } }
I20260925 02:53:05.507992 17790 raft_consensus.cc:374] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:05.508050 17790 raft_consensus.cc:729] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a896c0ddd8b4d6386e1227d9308ad90, State: Initialized, Role: FOLLOWER
I20260925 02:53:05.508284 17504 heartbeater.cc:503] Master 127.12.222.254:45313 was elected leader, sending a full tablet report...
I20260925 02:53:05.508265 17788 consensus_queue.cc:260] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a896c0ddd8b4d6386e1227d9308ad90" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 32885 } } peers { permanent_uuid: "b981e6f955d743158fdcd29082ec8f44" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38889 } } peers { permanent_uuid: "bd60d473ccb345cdb911cb6da271bdac" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 35817 } }
I20260925 02:53:05.508417 17789 ts_tablet_manager.cc:1428] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260925 02:53:05.508464 17790 consensus_queue.cc:260] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a896c0ddd8b4d6386e1227d9308ad90" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 32885 } } peers { permanent_uuid: "b981e6f955d743158fdcd29082ec8f44" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38889 } } peers { permanent_uuid: "bd60d473ccb345cdb911cb6da271bdac" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 35817 } }
I20260925 02:53:05.509068 17638 heartbeater.cc:503] Master 127.12.222.254:45313 was elected leader, sending a full tablet report...
I20260925 02:53:05.509111 17788 ts_tablet_manager.cc:1428] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:53:05.509192 17790 ts_tablet_manager.cc:1428] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:53:05.509371 17771 heartbeater.cc:503] Master 127.12.222.254:45313 was elected leader, sending a full tablet report...
W20260925 02:53:05.562738 17505 tablet.cc:2369] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:53:05.689157 17772 tablet.cc:2369] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:53:05.705350 17794 raft_consensus.cc:482] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:05.705578 17794 raft_consensus.cc:504] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a896c0ddd8b4d6386e1227d9308ad90" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 32885 } } peers { permanent_uuid: "b981e6f955d743158fdcd29082ec8f44" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38889 } } peers { permanent_uuid: "bd60d473ccb345cdb911cb6da271bdac" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 35817 } }
I20260925 02:53:05.706815 17794 leader_election.cc:288] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a896c0ddd8b4d6386e1227d9308ad90 (127.12.222.194:32885), b981e6f955d743158fdcd29082ec8f44 (127.12.222.195:38889)
I20260925 02:53:05.710716 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.707219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:37505 (local address 127.12.222.195:38889)
0925 02:53:05.707333 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:05.707337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:05.707598 (+ 261us) server_negotiation.cc:407] Connection header received
0925 02:53:05.707709 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:05.707711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:05.707777 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:05.707852 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:05.708180 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.708691 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.709798 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.710184 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.710292 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:05.710342 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:05.710537 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 02:53:05.710599 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 02:53:05.710731 17797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.707028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:32885 (local address 127.12.222.193:42997)
0925 02:53:05.707395 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:05.707407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:05.707510 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:05.707780 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:05.707784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:05.707802 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:05.707986 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:05.707993 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.708790 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.708796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:05.709675 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:05.709684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.710329 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.710332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:05.710432 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:05.710448 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:53:05.710533 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 02:53:05.710599 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":71,"thread_start_us":97,"threads_started":1}
I20260925 02:53:05.710724 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.707164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:42997 (local address 127.12.222.194:32885)
0925 02:53:05.707305 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:05.707310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:05.707497 (+ 187us) server_negotiation.cc:407] Connection header received
0925 02:53:05.707618 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:05.707621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:05.707678 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:05.707749 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:05.708102 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.708672 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.709801 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.710192 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.710323 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:05.710369 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:05.710565 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 02:53:05.710630 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:53:05.710738 17798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.707111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:38889 (local address 127.12.222.193:37505)
0925 02:53:05.707490 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:05.707503 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:05.707611 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:05.707885 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:05.707887 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:05.707901 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:05.708063 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:05.708070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.708836 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.708838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:05.709674 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:05.709683 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.710312 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.710315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:05.710410 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:05.710432 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:53:05.710494 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 02:53:05.710581 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":27,"thread_start_us":85,"threads_started":1}
I20260925 02:53:05.711448 17592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4996bb9b6ae24f9c90ed5a04350585de" candidate_uuid: "bd60d473ccb345cdb911cb6da271bdac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a896c0ddd8b4d6386e1227d9308ad90" is_pre_election: true
I20260925 02:53:05.711757 17592 raft_consensus.cc:2454] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd60d473ccb345cdb911cb6da271bdac in term 0.
I20260925 02:53:05.712003 17725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4996bb9b6ae24f9c90ed5a04350585de" candidate_uuid: "bd60d473ccb345cdb911cb6da271bdac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b981e6f955d743158fdcd29082ec8f44" is_pre_election: true
I20260925 02:53:05.712327 17725 raft_consensus.cc:2454] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd60d473ccb345cdb911cb6da271bdac in term 0.
I20260925 02:53:05.712332 17390 leader_election.cc:302] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a896c0ddd8b4d6386e1227d9308ad90, bd60d473ccb345cdb911cb6da271bdac; no voters:
I20260925 02:53:05.712586 17794 raft_consensus.cc:2781] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:53:05.712702 17794 raft_consensus.cc:482] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:53:05.712759 17794 raft_consensus.cc:3037] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:05.713728 17794 raft_consensus.cc:504] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a896c0ddd8b4d6386e1227d9308ad90" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 32885 } } peers { permanent_uuid: "b981e6f955d743158fdcd29082ec8f44" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38889 } } peers { permanent_uuid: "bd60d473ccb345cdb911cb6da271bdac" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 35817 } }
I20260925 02:53:05.714087 17794 leader_election.cc:288] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [CANDIDATE]: Term 1 election: Requested vote from peers 4a896c0ddd8b4d6386e1227d9308ad90 (127.12.222.194:32885), b981e6f955d743158fdcd29082ec8f44 (127.12.222.195:38889)
I20260925 02:53:05.714401 17592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4996bb9b6ae24f9c90ed5a04350585de" candidate_uuid: "bd60d473ccb345cdb911cb6da271bdac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a896c0ddd8b4d6386e1227d9308ad90"
I20260925 02:53:05.714511 17725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4996bb9b6ae24f9c90ed5a04350585de" candidate_uuid: "bd60d473ccb345cdb911cb6da271bdac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b981e6f955d743158fdcd29082ec8f44"
I20260925 02:53:05.714507 17592 raft_consensus.cc:3037] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:05.714646 17725 raft_consensus.cc:3037] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:05.715531 17592 raft_consensus.cc:2454] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd60d473ccb345cdb911cb6da271bdac in term 1.
I20260925 02:53:05.715675 17725 raft_consensus.cc:2454] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd60d473ccb345cdb911cb6da271bdac in term 1.
I20260925 02:53:05.715816 17390 leader_election.cc:302] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a896c0ddd8b4d6386e1227d9308ad90, bd60d473ccb345cdb911cb6da271bdac; no voters:
I20260925 02:53:05.715994 17794 raft_consensus.cc:2781] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:05.716204 17794 raft_consensus.cc:686] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [term 1 LEADER]: Becoming Leader. State: Replica: bd60d473ccb345cdb911cb6da271bdac, State: Running, Role: LEADER
I20260925 02:53:05.716506 17794 consensus_queue.cc:237] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a896c0ddd8b4d6386e1227d9308ad90" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 32885 } } peers { permanent_uuid: "b981e6f955d743158fdcd29082ec8f44" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38889 } } peers { permanent_uuid: "bd60d473ccb345cdb911cb6da271bdac" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 35817 } }
I20260925 02:53:05.719386 17310 catalog_manager.cc:5495] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac reported cstate change: term changed from 0 to 1, leader changed from <none> to bd60d473ccb345cdb911cb6da271bdac (127.12.222.193). New cstate: current_term: 1 leader_uuid: "bd60d473ccb345cdb911cb6da271bdac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a896c0ddd8b4d6386e1227d9308ad90" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 32885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b981e6f955d743158fdcd29082ec8f44" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd60d473ccb345cdb911cb6da271bdac" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 35817 } health_report { overall_health: HEALTHY } } }
W20260925 02:53:05.752991 17639 tablet.cc:2369] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:53:05.775979 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.772707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41072 (local address 127.12.222.193:35817)
0925 02:53:05.772863 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:05.772867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:05.772881 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:53:05.773014 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:05.773016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:05.773068 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:05.773180 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:53:05.773364 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.773880 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.774672 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:05.774882 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:05.774969 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:05.775228 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:53:05.775616 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:53:05.775706 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:05.775748 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 02:53:05.775796 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:53:05.781690 17725 raft_consensus.cc:1261] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44 [term 1 FOLLOWER]: Refusing update from remote peer bd60d473ccb345cdb911cb6da271bdac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:05.781704 17592 raft_consensus.cc:1261] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90 [term 1 FOLLOWER]: Refusing update from remote peer bd60d473ccb345cdb911cb6da271bdac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:05.782336 17794 consensus_queue.cc:1035] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a896c0ddd8b4d6386e1227d9308ad90" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 32885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:05.782501 17799 consensus_queue.cc:1035] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b981e6f955d743158fdcd29082ec8f44" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:05.789196 17802 mvcc.cc:205] Tried to move back new op lower bound from 7333088402555080704 to 7333088402296946688. Current Snapshot: MvccSnapshot[applied={T|T < 7333088402555080704}]
I20260925 02:53:05.790582 17804 mvcc.cc:205] Tried to move back new op lower bound from 7333088402555080704 to 7333088402296946688. Current Snapshot: MvccSnapshot[applied={T|T < 7333088402555080704}]
I20260925 02:53:05.792312 17803 mvcc.cc:205] Tried to move back new op lower bound from 7333088402555080704 to 7333088402296946688. Current Snapshot: MvccSnapshot[applied={T|T < 7333088402555080704}]
I20260925 02:53:05.797442 17312 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:05.797598 17312 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:05.799696 17312 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5355a596c6940bd9ee6e35a11cfe44d: Sending DeleteTablet for 3 replicas of tablet 4996bb9b6ae24f9c90ed5a04350585de
I20260925 02:53:05.800617 17571 tablet_service.cc:1514] Processing DeleteTablet for tablet 4996bb9b6ae24f9c90ed5a04350585de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:05 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:51864
I20260925 02:53:05.800693 17437 tablet_service.cc:1514] Processing DeleteTablet for tablet 4996bb9b6ae24f9c90ed5a04350585de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:05 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:41068
I20260925 02:53:05.801321 17705 tablet_service.cc:1514] Processing DeleteTablet for tablet 4996bb9b6ae24f9c90ed5a04350585de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:05 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:38716
I20260925 02:53:05.801687 17812 tablet_replica.cc:330] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac: stopping tablet replica
I20260925 02:53:05.801990 17812 raft_consensus.cc:2229] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:53:05.802310 17812 raft_consensus.cc:2258] T 4996bb9b6ae24f9c90ed5a04350585de P bd60d473ccb345cdb911cb6da271bdac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:05.802395 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 17375
I20260925 02:53:05.802544 17811 tablet_replica.cc:330] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90: stopping tablet replica
I20260925 02:53:05.802802 17811 raft_consensus.cc:2229] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:05.803350 17811 raft_consensus.cc:2258] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:05.804407 17813 tablet_replica.cc:330] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44: stopping tablet replica
I20260925 02:53:05.804703 17813 raft_consensus.cc:2229] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:05.804785 17811 ts_tablet_manager.cc:1905] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:05.805039 17813 raft_consensus.cc:2258] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:05.806304 17813 ts_tablet_manager.cc:1905] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:05.808027 17811 ts_tablet_manager.cc:1918] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:05.808157 17811 log.cc:1196] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/4996bb9b6ae24f9c90ed5a04350585de
I20260925 02:53:05.808494 17811 ts_tablet_manager.cc:1939] T 4996bb9b6ae24f9c90ed5a04350585de P 4a896c0ddd8b4d6386e1227d9308ad90: Deleting consensus metadata
I20260925 02:53:05.808924 17813 ts_tablet_manager.cc:1918] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:05.809057 17813 log.cc:1196] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/4996bb9b6ae24f9c90ed5a04350585de
I20260925 02:53:05.809383 17813 ts_tablet_manager.cc:1939] T 4996bb9b6ae24f9c90ed5a04350585de P b981e6f955d743158fdcd29082ec8f44: Deleting consensus metadata
I20260925 02:53:05.809664 17296 catalog_manager.cc:4841] TS 4a896c0ddd8b4d6386e1227d9308ad90 (127.12.222.194:32885): tablet 4996bb9b6ae24f9c90ed5a04350585de (table test-table [id=ea60904927ef45e38d6d38463b3b11c6]) successfully deleted
I20260925 02:53:05.810448 17299 catalog_manager.cc:4841] TS b981e6f955d743158fdcd29082ec8f44 (127.12.222.195:38889): tablet 4996bb9b6ae24f9c90ed5a04350585de (table test-table [id=ea60904927ef45e38d6d38463b3b11c6]) successfully deleted
W20260925 02:53:05.811283 17298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.222.193:35817 (error 108)
I20260925 02:53:05.812062 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 17509
I20260925 02:53:05.812265 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:05.811944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:35817 (local address 127.0.0.1:41078)
0925 02:53:05.812077 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:05.812157 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.193:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 02:53:05.812696 17298 catalog_manager.cc:4584] TS bd60d473ccb345cdb911cb6da271bdac (127.12.222.193:35817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4996bb9b6ae24f9c90ed5a04350585de: Network error: Client connection negotiation failed: client connection to 127.12.222.193:35817: connect: Connection refused (error 111)
I20260925 02:53:05.817675 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 17643
I20260925 02:53:05.822978 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 17281
2026-09-25T02:53:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2287 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T02:53:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:05Z Disabled control of system clock
I20260925 02:53:05.852129 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:46153
--webserver_interface=127.12.222.254
--webserver_port=0
--builtin_ntp_servers=127.12.222.212:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:46153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 02:53:05.953588 17818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:05.953825 17818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:05.953902 17818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:05.957631 17818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:05.957675 17818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:05.957690 17818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:05.957703 17818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:05.957737 17818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:05.961956 17818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:46153
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:46153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:05.962848 17818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:05.964147 17818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:05.971783 17823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:05.971808 17818 server_base.cc:1034] running on GCE node
W20260925 02:53:05.971776 17824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:05.971776 17826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:05.972627 17818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:05.973553 17818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:05.974735 17818 hybrid_clock.cc:648] HybridClock initialized: now 1790304785974705 us; error 38 us; skew 500 ppm
I20260925 02:53:05.976496 17818 webserver.cc:460] Webserver started at http://127.12.222.254:42451/ using document root <none> and password file <none>
I20260925 02:53:05.976977 17818 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:05.977043 17818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:05.977182 17818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:05.978888 17818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "14d05fc16a174d7d8793be01af6c7280"
format_stamp: "Formatted at 2026-09-25 02:53:05 on dist-test-slave-252r"
server_key: "b7edf1e165ad6b9080a967a30754d6d0"
server_key_iv: "8aafce8b45f758c6f50675472ce9f6c7"
server_key_version: "encryptionkey@0"
I20260925 02:53:05.979353 17818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "14d05fc16a174d7d8793be01af6c7280"
format_stamp: "Formatted at 2026-09-25 02:53:05 on dist-test-slave-252r"
server_key: "b7edf1e165ad6b9080a967a30754d6d0"
server_key_iv: "8aafce8b45f758c6f50675472ce9f6c7"
server_key_version: "encryptionkey@0"
I20260925 02:53:05.982594 17818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 02:53:05.984697 17832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:05.985813 17818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:05.985941 17818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "14d05fc16a174d7d8793be01af6c7280"
format_stamp: "Formatted at 2026-09-25 02:53:05 on dist-test-slave-252r"
server_key: "b7edf1e165ad6b9080a967a30754d6d0"
server_key_iv: "8aafce8b45f758c6f50675472ce9f6c7"
server_key_version: "encryptionkey@0"
I20260925 02:53:05.986038 17818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:05.990095 17818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:05.990602 17818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:05.990762 17818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:05.997061 17818 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:46153
I20260925 02:53:05.997082 17884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:46153 every 8 connection(s)
I20260925 02:53:05.997853 17818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:06.001950 17885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:06.006834 17885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280: Bootstrap starting.
I20260925 02:53:06.006978 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 17818
I20260925 02:53:06.007092 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:53:06.007354 13179 external_mini_cluster.cc:1428] Setting key 9dc7dbcb4f8741baaa834d892d7efcfa
I20260925 02:53:06.009429 17885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:06.010255 17885 log.cc:824] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:06.012311 17885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280: No bootstrap required, opened a new log
I20260925 02:53:06.013083 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.008666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32816 (local address 127.12.222.254:46153)
0925 02:53:06.009283 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:06.009308 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:06.009378 (+ 70us) server_negotiation.cc:407] Connection header received
0925 02:53:06.010197 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:06.010220 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:06.010650 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:06.010963 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:06.011686 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:06.011691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:06.011704 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:06.011715 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:06.011776 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:06.011797 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:06.011900 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:06.011972 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 02:53:06.012186 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
I20260925 02:53:06.016040 17885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d05fc16a174d7d8793be01af6c7280" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 46153 } }
I20260925 02:53:06.016217 17885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:06.016315 17885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d05fc16a174d7d8793be01af6c7280, State: Initialized, Role: FOLLOWER
I20260925 02:53:06.016754 17885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d05fc16a174d7d8793be01af6c7280" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 46153 } }
I20260925 02:53:06.016870 17885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:06.016911 17885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:06.016983 17885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:06.017679 17885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d05fc16a174d7d8793be01af6c7280" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 46153 } }
I20260925 02:53:06.017937 17885 leader_election.cc:302] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14d05fc16a174d7d8793be01af6c7280; no voters:
I20260925 02:53:06.018160 17885 leader_election.cc:288] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:53:06.018288 17890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:06.018503 17890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280 [term 1 LEADER]: Becoming Leader. State: Replica: 14d05fc16a174d7d8793be01af6c7280, State: Running, Role: LEADER
I20260925 02:53:06.018793 17890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d05fc16a174d7d8793be01af6c7280" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 46153 } }
I20260925 02:53:06.019328 17885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:06.020316 17892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14d05fc16a174d7d8793be01af6c7280. Latest consensus state: current_term: 1 leader_uuid: "14d05fc16a174d7d8793be01af6c7280" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d05fc16a174d7d8793be01af6c7280" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 46153 } } }
I20260925 02:53:06.020438 17892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:06.020780 17899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:06.020788 17891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14d05fc16a174d7d8793be01af6c7280" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d05fc16a174d7d8793be01af6c7280" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 46153 } } }
I20260925 02:53:06.021077 17891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:06.023267 17899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:06.028443 17899 catalog_manager.cc:1347] Generated new cluster ID: cd236b7c5302490e88d825b0a1ef0f42
I20260925 02:53:06.028497 17899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:06.042384 17899 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:06.042637 17899 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:06.048461 17899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280: Generated new TSK 0
I20260925 02:53:06.048907 17899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:53:06.059311 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:46153
--builtin_ntp_servers=127.12.222.212:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 02:53:06.160041 17910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:06.160264 17910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:06.160297 17910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:06.163944 17910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:06.163983 17910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:06.164034 17910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:53:06.168126 17910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:46153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:06.169018 17910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:06.170223 17910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:06.179054 17915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:06.179085 17918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:06.179139 17910 server_base.cc:1034] running on GCE node
W20260925 02:53:06.179054 17916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:06.179569 17910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:06.180094 17910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:06.181265 17910 hybrid_clock.cc:648] HybridClock initialized: now 1790304786181251 us; error 31 us; skew 500 ppm
I20260925 02:53:06.182843 17910 webserver.cc:460] Webserver started at http://127.12.222.193:44741/ using document root <none> and password file <none>
I20260925 02:53:06.183336 17910 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:06.183425 17910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:06.183625 17910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:06.185549 17910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "8e712a1c17644dfc97c4397ec3e10da0"
format_stamp: "Formatted at 2026-09-25 02:53:06 on dist-test-slave-252r"
server_key: "e1c6d56728ead7bab5b9c7b742b67d00"
server_key_iv: "79eef4764b3b32478ef50a15c86c91bf"
server_key_version: "encryptionkey@0"
I20260925 02:53:06.186064 17910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "8e712a1c17644dfc97c4397ec3e10da0"
format_stamp: "Formatted at 2026-09-25 02:53:06 on dist-test-slave-252r"
server_key: "e1c6d56728ead7bab5b9c7b742b67d00"
server_key_iv: "79eef4764b3b32478ef50a15c86c91bf"
server_key_version: "encryptionkey@0"
I20260925 02:53:06.189169 17910 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 02:53:06.191327 17924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:06.192535 17910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:53:06.192668 17910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "8e712a1c17644dfc97c4397ec3e10da0"
format_stamp: "Formatted at 2026-09-25 02:53:06 on dist-test-slave-252r"
server_key: "e1c6d56728ead7bab5b9c7b742b67d00"
server_key_iv: "79eef4764b3b32478ef50a15c86c91bf"
server_key_version: "encryptionkey@0"
I20260925 02:53:06.192782 17910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:06.196858 17910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:06.197402 17910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:06.197557 17910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:06.198014 17910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:06.198765 17910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:06.198827 17910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:06.198881 17910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:06.198918 17910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:06.206911 17910 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:45049
I20260925 02:53:06.206934 18038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:45049 every 8 connection(s)
I20260925 02:53:06.207667 17910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:53:06.212994 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.209233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:49055 (local address 127.12.222.254:46153)
0925 02:53:06.209359 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:06.209362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:06.210240 (+ 878us) server_negotiation.cc:407] Connection header received
0925 02:53:06.211415 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:06.211418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:06.211474 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:06.211578 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:06.212562 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:06.212565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:06.212568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:06.212579 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:06.212600 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:06.212613 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:06.212699 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:06.212833 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 02:53:06.212875 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:53:06.213779 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.209547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:46153 (local address 127.12.222.193:49055)
0925 02:53:06.210113 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:06.210147 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:06.211203 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:06.211729 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:06.211735 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:06.211815 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:06.212330 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:53:06.212332 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:06.212365 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:53:06.212424 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:06.212695 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:06.212704 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:06.212711 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:53:06.212872 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 02:53:06.213098 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":102,"threads_started":1}
I20260925 02:53:06.214756 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 17910
I20260925 02:53:06.214880 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:53:06.214962 18039 heartbeater.cc:347] Connected to a master server at 127.12.222.254:46153
I20260925 02:53:06.215180 18039 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:06.215233 13179 external_mini_cluster.cc:1428] Setting key cbecff4d02c0fd909f93ed9d689c572a
I20260925 02:53:06.215561 18039 heartbeater.cc:511] Master 127.12.222.254:46153 requested a full tablet report, sending...
I20260925 02:53:06.216980 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:46153
--builtin_ntp_servers=127.12.222.212:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 02:53:06.216996 17849 ts_manager.cc:194] Registered new tserver with Master: 8e712a1c17644dfc97c4397ec3e10da0 (127.12.222.193:45049)
W20260925 02:53:06.321280 18042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:06.321506 18042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:06.321542 18042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:06.325284 18042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:06.325325 18042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:06.325371 18042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:53:06.329533 18042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:46153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:06.330369 18042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:06.331538 18042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:06.339955 18048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:06.339991 18050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:06.339982 18047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:06.340107 18042 server_base.cc:1034] running on GCE node
I20260925 02:53:06.340567 18042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:06.341181 18042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:06.342371 18042 hybrid_clock.cc:648] HybridClock initialized: now 1790304786342362 us; error 29 us; skew 500 ppm
I20260925 02:53:06.344099 18042 webserver.cc:460] Webserver started at http://127.12.222.194:33217/ using document root <none> and password file <none>
I20260925 02:53:06.344673 18042 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:06.344766 18042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:06.344971 18042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:06.346755 18042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "adaa572ae8ca4c70bde845b74d200452"
format_stamp: "Formatted at 2026-09-25 02:53:06 on dist-test-slave-252r"
server_key: "24a2fbb74b00eaa5f1c5ff6407837248"
server_key_iv: "886c9143ae4c7947325179c0353f7771"
server_key_version: "encryptionkey@0"
I20260925 02:53:06.347257 18042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "adaa572ae8ca4c70bde845b74d200452"
format_stamp: "Formatted at 2026-09-25 02:53:06 on dist-test-slave-252r"
server_key: "24a2fbb74b00eaa5f1c5ff6407837248"
server_key_iv: "886c9143ae4c7947325179c0353f7771"
server_key_version: "encryptionkey@0"
I20260925 02:53:06.350689 18042 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:53:06.353041 18056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:06.354094 18042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 02:53:06.354218 18042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "adaa572ae8ca4c70bde845b74d200452"
format_stamp: "Formatted at 2026-09-25 02:53:06 on dist-test-slave-252r"
server_key: "24a2fbb74b00eaa5f1c5ff6407837248"
server_key_iv: "886c9143ae4c7947325179c0353f7771"
server_key_version: "encryptionkey@0"
I20260925 02:53:06.354321 18042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:06.358312 18042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:06.358820 18042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:06.358983 18042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:06.359445 18042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:06.360203 18042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:06.360293 18042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:06.360330 18042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:06.360354 18042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:06.367831 18042 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:36961
I20260925 02:53:06.367897 18169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:36961 every 8 connection(s)
I20260925 02:53:06.368660 18042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:53:06.372202 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 18042
I20260925 02:53:06.372365 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:53:06.372619 13179 external_mini_cluster.cc:1428] Setting key 0e88d19d612ac08fdbefd54e2da95862
I20260925 02:53:06.374102 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.370208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:60353 (local address 127.12.222.254:46153)
0925 02:53:06.370326 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:06.370329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:06.371228 (+ 899us) server_negotiation.cc:407] Connection header received
0925 02:53:06.372398 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:06.372402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:06.372445 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:06.372542 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:06.373615 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:06.373619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:06.373621 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:06.373642 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:06.373660 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:06.373668 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:06.373777 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:06.373916 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 02:53:06.373967 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:53:06.374717 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:46153
--builtin_ntp_servers=127.12.222.212:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 02:53:06.374917 18172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.370541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:46153 (local address 127.12.222.194:60353)
0925 02:53:06.371098 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:06.371133 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:06.372148 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:06.372714 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:06.372724 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:06.372818 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:06.373359 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:53:06.373361 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:06.373405 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:53:06.373475 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:06.373765 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:06.373774 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:06.373780 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:53:06.373958 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 02:53:06.374187 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":100,"threads_started":1}
I20260925 02:53:06.376039 18170 heartbeater.cc:347] Connected to a master server at 127.12.222.254:46153
I20260925 02:53:06.376281 18170 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:06.376717 18170 heartbeater.cc:511] Master 127.12.222.254:46153 requested a full tablet report, sending...
I20260925 02:53:06.377693 17849 ts_manager.cc:194] Registered new tserver with Master: adaa572ae8ca4c70bde845b74d200452 (127.12.222.194:36961)
W20260925 02:53:06.476451 18173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:06.476649 18173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:06.476683 18173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:06.480280 18173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:06.480322 18173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:06.480373 18173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:53:06.484602 18173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:46153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:06.485373 18173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:06.486472 18173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:06.494853 18178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:06.494859 18179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:06.495049 18173 server_base.cc:1034] running on GCE node
W20260925 02:53:06.494868 18181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:06.495527 18173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:06.496115 18173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:06.497267 18173 hybrid_clock.cc:648] HybridClock initialized: now 1790304786497223 us; error 75 us; skew 500 ppm
I20260925 02:53:06.498843 18173 webserver.cc:460] Webserver started at http://127.12.222.195:44837/ using document root <none> and password file <none>
I20260925 02:53:06.499346 18173 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:06.499401 18173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:06.499629 18173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:06.501545 18173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "fa60c1cf54ed4395a7dfeee04a18c81b"
format_stamp: "Formatted at 2026-09-25 02:53:06 on dist-test-slave-252r"
server_key: "1839ae38c08a95c3136a8b6cac1d231e"
server_key_iv: "3bfdaac209605ae8cf10551697ee5211"
server_key_version: "encryptionkey@0"
I20260925 02:53:06.502104 18173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "fa60c1cf54ed4395a7dfeee04a18c81b"
format_stamp: "Formatted at 2026-09-25 02:53:06 on dist-test-slave-252r"
server_key: "1839ae38c08a95c3136a8b6cac1d231e"
server_key_iv: "3bfdaac209605ae8cf10551697ee5211"
server_key_version: "encryptionkey@0"
I20260925 02:53:06.505604 18173 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 02:53:06.507795 18187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:06.508912 18173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:06.509043 18173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "fa60c1cf54ed4395a7dfeee04a18c81b"
format_stamp: "Formatted at 2026-09-25 02:53:06 on dist-test-slave-252r"
server_key: "1839ae38c08a95c3136a8b6cac1d231e"
server_key_iv: "3bfdaac209605ae8cf10551697ee5211"
server_key_version: "encryptionkey@0"
I20260925 02:53:06.509145 18173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:06.513190 18173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:06.513757 18173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:06.513931 18173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:06.514421 18173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:06.515271 18173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:06.515339 18173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:06.515403 18173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:06.515461 18173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:06.524163 18173 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:36305
I20260925 02:53:06.524194 18300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:36305 every 8 connection(s)
I20260925 02:53:06.524978 18173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:53:06.530297 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.526464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:36101 (local address 127.12.222.254:46153)
0925 02:53:06.526589 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:06.526594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:06.527552 (+ 958us) server_negotiation.cc:407] Connection header received
0925 02:53:06.528821 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:06.528824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:06.528859 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:06.528928 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:06.529949 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:06.529951 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:06.529953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:06.529962 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:06.529976 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:06.529982 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:06.530046 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:06.530168 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 02:53:06.530201 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:53:06.530939 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 18173
I20260925 02:53:06.531059 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:53:06.531251 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.526826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:46153 (local address 127.12.222.195:36101)
0925 02:53:06.527421 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:06.527458 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:06.528644 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:06.529082 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:06.529088 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:06.529164 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:06.529728 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:53:06.529729 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:06.529763 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:53:06.529824 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:06.530052 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:06.530058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:06.530062 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:53:06.530185 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 02:53:06.530348 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":115,"threads_started":1}
I20260925 02:53:06.531389 13179 external_mini_cluster.cc:1428] Setting key 32138412eaa0bfe93940a14686370934
I20260925 02:53:06.532400 18301 heartbeater.cc:347] Connected to a master server at 127.12.222.254:46153
I20260925 02:53:06.532624 18301 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:06.533006 18301 heartbeater.cc:511] Master 127.12.222.254:46153 requested a full tablet report, sending...
I20260925 02:53:06.534013 17849 ts_manager.cc:194] Registered new tserver with Master: fa60c1cf54ed4395a7dfeee04a18c81b (127.12.222.195:36305)
I20260925 02:53:06.545018 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:53:06.548039 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.546781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.12.222.254:46153)
0925 02:53:06.546948 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:06.546951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:06.547031 (+ 80us) server_negotiation.cc:407] Connection header received
0925 02:53:06.547130 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:06.547132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:06.547160 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:06.547220 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:06.547751 (+ 531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:06.547754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:06.547756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:06.547765 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:06.547780 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:06.547786 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:06.547847 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:06.547885 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 02:53:06.547912 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 02:53:06.550647 17849 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:53:06.553622 17849 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:53:06.564352 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.561386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:36305 (local address 127.0.0.1:55648)
0925 02:53:06.561866 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:06.561908 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:06.562002 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:06.562878 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:06.562883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:06.562962 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:06.563488 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:53:06.563492 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:06.563516 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:53:06.563585 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:06.564170 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:06.564173 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:06.564176 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:53:06.564223 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 02:53:06.564271 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":82,"threads_started":1}
I20260925 02:53:06.564371 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.562144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55894 (local address 127.12.222.194:36961)
0925 02:53:06.562784 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:06.562795 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:06.562818 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:53:06.562870 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:06.562875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:06.563051 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:06.563190 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:06.563804 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:06.563809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:06.563817 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:06.563824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:06.563851 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:06.563864 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:06.563949 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:06.564133 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 02:53:06.564203 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":129,"threads_started":1}
I20260925 02:53:06.564334 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.561483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:36961 (local address 127.0.0.1:55894)
0925 02:53:06.562211 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:06.562224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:06.562290 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:06.563178 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:06.563180 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:06.563197 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:06.563653 (+ 456us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:53:06.563664 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:06.563678 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:53:06.563713 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:06.563964 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:06.563973 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:06.563979 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:53:06.564098 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 02:53:06.564170 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":79,"threads_started":1}
I20260925 02:53:06.565006 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.561444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.12.222.195:36305)
0925 02:53:06.562361 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:06.562373 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:06.562398 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:53:06.562471 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:06.562480 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:06.562690 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:06.562885 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:06.563981 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:06.563986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:06.563996 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:06.564006 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:06.564043 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:06.564071 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:06.564732 (+ 661us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:06.564785 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 02:53:06.564855 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":82,"threads_started":1}
I20260925 02:53:06.566176 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.561821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:45049 (local address 127.0.0.1:41600)
0925 02:53:06.562430 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:06.562443 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:06.562507 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:06.565133 (+ 2626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:06.565135 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:06.565148 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:06.565543 (+ 395us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:53:06.565545 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:06.565555 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:53:06.565579 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:06.565928 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:06.565931 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:06.565934 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:53:06.566011 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:53:06.566045 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":113,"threads_started":1}
I20260925 02:53:06.566538 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.562300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.12.222.193:45049)
0925 02:53:06.564155 (+ 1855us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:06.564174 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:06.564195 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:53:06.564273 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:06.564278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:06.564435 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:06.564569 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:06.565713 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:06.565718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:06.565728 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:06.565739 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:06.565776 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:06.565805 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:06.565890 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:06.566337 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 02:53:06.566404 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1727,"thread_start_us":145,"threads_started":1}
I20260925 02:53:06.567544 18104 tablet_service.cc:1467] Processing CreateTablet for tablet 55d36829b0c3473689fcba7943aaace0 (DEFAULT_TABLE table=test-table [id=8c2d3f4409cd4da9bd534c02a6814621]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:06.567690 18235 tablet_service.cc:1467] Processing CreateTablet for tablet 55d36829b0c3473689fcba7943aaace0 (DEFAULT_TABLE table=test-table [id=8c2d3f4409cd4da9bd534c02a6814621]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:06.568514 18235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d36829b0c3473689fcba7943aaace0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:06.568533 18104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d36829b0c3473689fcba7943aaace0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:06.568902 17973 tablet_service.cc:1467] Processing CreateTablet for tablet 55d36829b0c3473689fcba7943aaace0 (DEFAULT_TABLE table=test-table [id=8c2d3f4409cd4da9bd534c02a6814621]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:06.569742 17973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d36829b0c3473689fcba7943aaace0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:06.574097 18318 tablet_bootstrap.cc:492] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b: Bootstrap starting.
I20260925 02:53:06.575662 18319 tablet_bootstrap.cc:492] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452: Bootstrap starting.
I20260925 02:53:06.576258 18318 tablet_bootstrap.cc:654] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:06.576568 18320 tablet_bootstrap.cc:492] T 55d36829b0c3473689fcba7943aaace0 P 8e712a1c17644dfc97c4397ec3e10da0: Bootstrap starting.
I20260925 02:53:06.577148 18318 log.cc:824] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:06.577801 18319 tablet_bootstrap.cc:654] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:06.578480 18320 tablet_bootstrap.cc:654] T 55d36829b0c3473689fcba7943aaace0 P 8e712a1c17644dfc97c4397ec3e10da0: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:06.578575 18319 log.cc:824] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:06.579124 18318 tablet_bootstrap.cc:492] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b: No bootstrap required, opened a new log
I20260925 02:53:06.579270 18320 log.cc:824] T 55d36829b0c3473689fcba7943aaace0 P 8e712a1c17644dfc97c4397ec3e10da0: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:06.579340 18318 ts_tablet_manager.cc:1397] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 02:53:06.580513 18319 tablet_bootstrap.cc:492] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452: No bootstrap required, opened a new log
I20260925 02:53:06.580673 18319 ts_tablet_manager.cc:1397] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:53:06.580891 18320 tablet_bootstrap.cc:492] T 55d36829b0c3473689fcba7943aaace0 P 8e712a1c17644dfc97c4397ec3e10da0: No bootstrap required, opened a new log
I20260925 02:53:06.581059 18320 ts_tablet_manager.cc:1397] T 55d36829b0c3473689fcba7943aaace0 P 8e712a1c17644dfc97c4397ec3e10da0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:53:06.583488 18318 raft_consensus.cc:348] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60c1cf54ed4395a7dfeee04a18c81b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36305 } } peers { permanent_uuid: "adaa572ae8ca4c70bde845b74d200452" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36961 } } peers { permanent_uuid: "8e712a1c17644dfc97c4397ec3e10da0" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 45049 } }
I20260925 02:53:06.583782 18318 raft_consensus.cc:374] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:06.583838 18318 raft_consensus.cc:729] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa60c1cf54ed4395a7dfeee04a18c81b, State: Initialized, Role: FOLLOWER
I20260925 02:53:06.584280 18318 consensus_queue.cc:260] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60c1cf54ed4395a7dfeee04a18c81b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36305 } } peers { permanent_uuid: "adaa572ae8ca4c70bde845b74d200452" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36961 } } peers { permanent_uuid: "8e712a1c17644dfc97c4397ec3e10da0" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 45049 } }
I20260925 02:53:06.584455 18320 raft_consensus.cc:348] T 55d36829b0c3473689fcba7943aaace0 P 8e712a1c17644dfc97c4397ec3e10da0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60c1cf54ed4395a7dfeee04a18c81b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36305 } } peers { permanent_uuid: "adaa572ae8ca4c70bde845b74d200452" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36961 } } peers { permanent_uuid: "8e712a1c17644dfc97c4397ec3e10da0" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 45049 } }
I20260925 02:53:06.584705 18320 raft_consensus.cc:374] T 55d36829b0c3473689fcba7943aaace0 P 8e712a1c17644dfc97c4397ec3e10da0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:06.584764 18320 raft_consensus.cc:729] T 55d36829b0c3473689fcba7943aaace0 P 8e712a1c17644dfc97c4397ec3e10da0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e712a1c17644dfc97c4397ec3e10da0, State: Initialized, Role: FOLLOWER
I20260925 02:53:06.584772 18319 raft_consensus.cc:348] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60c1cf54ed4395a7dfeee04a18c81b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36305 } } peers { permanent_uuid: "adaa572ae8ca4c70bde845b74d200452" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36961 } } peers { permanent_uuid: "8e712a1c17644dfc97c4397ec3e10da0" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 45049 } }
I20260925 02:53:06.585001 18319 raft_consensus.cc:374] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:06.585043 18319 raft_consensus.cc:729] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adaa572ae8ca4c70bde845b74d200452, State: Initialized, Role: FOLLOWER
I20260925 02:53:06.585043 18301 heartbeater.cc:503] Master 127.12.222.254:46153 was elected leader, sending a full tablet report...
I20260925 02:53:06.585243 18318 ts_tablet_manager.cc:1428] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260925 02:53:06.585201 18320 consensus_queue.cc:260] T 55d36829b0c3473689fcba7943aaace0 P 8e712a1c17644dfc97c4397ec3e10da0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60c1cf54ed4395a7dfeee04a18c81b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36305 } } peers { permanent_uuid: "adaa572ae8ca4c70bde845b74d200452" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36961 } } peers { permanent_uuid: "8e712a1c17644dfc97c4397ec3e10da0" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 45049 } }
I20260925 02:53:06.585436 18319 consensus_queue.cc:260] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60c1cf54ed4395a7dfeee04a18c81b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36305 } } peers { permanent_uuid: "adaa572ae8ca4c70bde845b74d200452" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36961 } } peers { permanent_uuid: "8e712a1c17644dfc97c4397ec3e10da0" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 45049 } }
I20260925 02:53:06.585968 18039 heartbeater.cc:503] Master 127.12.222.254:46153 was elected leader, sending a full tablet report...
I20260925 02:53:06.586076 18320 ts_tablet_manager.cc:1428] T 55d36829b0c3473689fcba7943aaace0 P 8e712a1c17644dfc97c4397ec3e10da0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 02:53:06.586483 18170 heartbeater.cc:503] Master 127.12.222.254:46153 was elected leader, sending a full tablet report...
I20260925 02:53:06.586962 18319 ts_tablet_manager.cc:1428] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260925 02:53:06.619668 18171 tablet.cc:2369] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:53:06.708791 18040 tablet.cc:2369] T 55d36829b0c3473689fcba7943aaace0 P 8e712a1c17644dfc97c4397ec3e10da0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:53:06.775801 18302 tablet.cc:2369] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:53:06.820199 18326 raft_consensus.cc:482] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:06.820461 18326 raft_consensus.cc:504] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60c1cf54ed4395a7dfeee04a18c81b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36305 } } peers { permanent_uuid: "adaa572ae8ca4c70bde845b74d200452" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36961 } } peers { permanent_uuid: "8e712a1c17644dfc97c4397ec3e10da0" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 45049 } }
I20260925 02:53:06.821769 18326 leader_election.cc:288] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa60c1cf54ed4395a7dfeee04a18c81b (127.12.222.195:36305), 8e712a1c17644dfc97c4397ec3e10da0 (127.12.222.193:45049)
I20260925 02:53:06.823688 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.822142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:50209 (local address 127.12.222.193:45049)
0925 02:53:06.822263 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:06.822268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:06.822523 (+ 255us) server_negotiation.cc:407] Connection header received
0925 02:53:06.822661 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:06.822664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:06.822697 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:06.822773 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:06.823299 (+ 526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:06.823302 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:06.823304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:06.823315 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:06.823334 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:06.823342 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:06.823409 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:06.823514 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 02:53:06.823558 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:53:06.823700 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.822172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:40585 (local address 127.12.222.195:36305)
0925 02:53:06.822295 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:06.822299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:06.822311 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:53:06.822359 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:06.822361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:06.822389 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:06.822472 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:06.823151 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:06.823154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:06.823156 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:06.823169 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:06.823191 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:06.823200 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:06.823293 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:06.823546 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 02:53:06.823583 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:53:06.823699 18172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.821966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:36305 (local address 127.12.222.194:40585)
0925 02:53:06.822115 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:06.822129 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:06.822214 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:06.822491 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:06.822493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:06.822507 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:06.822979 (+ 472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:53:06.822982 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:06.822994 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:53:06.823026 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:06.823444 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:06.823446 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:06.823448 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:53:06.823538 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 02:53:06.823572 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 02:53:06.823712 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.822049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:45049 (local address 127.12.222.194:50209)
0925 02:53:06.822424 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:06.822434 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:06.822550 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:06.822789 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:06.822792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:06.822804 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:06.823170 (+ 366us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:53:06.823172 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:06.823182 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:53:06.823204 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:06.823411 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:06.823414 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:06.823416 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:53:06.823503 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 02:53:06.823540 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":97,"threads_started":1}
I20260925 02:53:06.824478 18255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d36829b0c3473689fcba7943aaace0" candidate_uuid: "adaa572ae8ca4c70bde845b74d200452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60c1cf54ed4395a7dfeee04a18c81b" is_pre_election: true
I20260925 02:53:06.824801 18255 raft_consensus.cc:2454] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adaa572ae8ca4c70bde845b74d200452 in term 0.
I20260925 02:53:06.824802 17993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d36829b0c3473689fcba7943aaace0" candidate_uuid: "adaa572ae8ca4c70bde845b74d200452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e712a1c17644dfc97c4397ec3e10da0" is_pre_election: true
I20260925 02:53:06.825121 17993 raft_consensus.cc:2454] T 55d36829b0c3473689fcba7943aaace0 P 8e712a1c17644dfc97c4397ec3e10da0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adaa572ae8ca4c70bde845b74d200452 in term 0.
I20260925 02:53:06.825276 18058 leader_election.cc:302] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adaa572ae8ca4c70bde845b74d200452, fa60c1cf54ed4395a7dfeee04a18c81b; no voters:
I20260925 02:53:06.825527 18326 raft_consensus.cc:2781] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:53:06.825603 18326 raft_consensus.cc:482] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:53:06.825630 18326 raft_consensus.cc:3037] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:06.826591 18326 raft_consensus.cc:504] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60c1cf54ed4395a7dfeee04a18c81b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36305 } } peers { permanent_uuid: "adaa572ae8ca4c70bde845b74d200452" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36961 } } peers { permanent_uuid: "8e712a1c17644dfc97c4397ec3e10da0" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 45049 } }
I20260925 02:53:06.826890 18326 leader_election.cc:288] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [CANDIDATE]: Term 1 election: Requested vote from peers fa60c1cf54ed4395a7dfeee04a18c81b (127.12.222.195:36305), 8e712a1c17644dfc97c4397ec3e10da0 (127.12.222.193:45049)
I20260925 02:53:06.827217 17993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d36829b0c3473689fcba7943aaace0" candidate_uuid: "adaa572ae8ca4c70bde845b74d200452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e712a1c17644dfc97c4397ec3e10da0"
I20260925 02:53:06.827224 18255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d36829b0c3473689fcba7943aaace0" candidate_uuid: "adaa572ae8ca4c70bde845b74d200452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa60c1cf54ed4395a7dfeee04a18c81b"
I20260925 02:53:06.827342 18255 raft_consensus.cc:3037] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:06.827343 17993 raft_consensus.cc:3037] T 55d36829b0c3473689fcba7943aaace0 P 8e712a1c17644dfc97c4397ec3e10da0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:06.828567 18255 raft_consensus.cc:2454] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adaa572ae8ca4c70bde845b74d200452 in term 1.
I20260925 02:53:06.828567 17993 raft_consensus.cc:2454] T 55d36829b0c3473689fcba7943aaace0 P 8e712a1c17644dfc97c4397ec3e10da0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adaa572ae8ca4c70bde845b74d200452 in term 1.
I20260925 02:53:06.828850 18059 leader_election.cc:302] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e712a1c17644dfc97c4397ec3e10da0, adaa572ae8ca4c70bde845b74d200452; no voters:
I20260925 02:53:06.828992 18326 raft_consensus.cc:2781] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:06.829219 18326 raft_consensus.cc:686] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [term 1 LEADER]: Becoming Leader. State: Replica: adaa572ae8ca4c70bde845b74d200452, State: Running, Role: LEADER
I20260925 02:53:06.829523 18326 consensus_queue.cc:237] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60c1cf54ed4395a7dfeee04a18c81b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36305 } } peers { permanent_uuid: "adaa572ae8ca4c70bde845b74d200452" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36961 } } peers { permanent_uuid: "8e712a1c17644dfc97c4397ec3e10da0" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 45049 } }
I20260925 02:53:06.832552 17848 catalog_manager.cc:5495] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 reported cstate change: term changed from 0 to 1, leader changed from <none> to adaa572ae8ca4c70bde845b74d200452 (127.12.222.194). New cstate: current_term: 1 leader_uuid: "adaa572ae8ca4c70bde845b74d200452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa60c1cf54ed4395a7dfeee04a18c81b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adaa572ae8ca4c70bde845b74d200452" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e712a1c17644dfc97c4397ec3e10da0" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 45049 } health_report { overall_health: UNKNOWN } } }
I20260925 02:53:06.853399 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.851847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.12.222.194:36961)
0925 02:53:06.851959 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:06.851964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:06.852047 (+ 83us) server_negotiation.cc:407] Connection header received
0925 02:53:06.852217 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:06.852246 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:06.852296 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:06.852378 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:06.853044 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:06.853047 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:06.853049 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:06.853060 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:06.853077 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:06.853086 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:06.853175 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:06.853249 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 02:53:06.853284 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:53:06.858830 17993 raft_consensus.cc:1261] T 55d36829b0c3473689fcba7943aaace0 P 8e712a1c17644dfc97c4397ec3e10da0 [term 1 FOLLOWER]: Refusing update from remote peer adaa572ae8ca4c70bde845b74d200452: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:06.858830 18255 raft_consensus.cc:1261] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b [term 1 FOLLOWER]: Refusing update from remote peer adaa572ae8ca4c70bde845b74d200452: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:06.859561 18328 consensus_queue.cc:1035] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e712a1c17644dfc97c4397ec3e10da0" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 45049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:06.859782 18326 consensus_queue.cc:1035] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa60c1cf54ed4395a7dfeee04a18c81b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:06.865875 18331 mvcc.cc:205] Tried to move back new op lower bound from 7333088406966951936 to 7333088406856437760. Current Snapshot: MvccSnapshot[applied={T|T < 7333088406966951936}]
I20260925 02:53:06.866109 18333 mvcc.cc:205] Tried to move back new op lower bound from 7333088406966951936 to 7333088406856437760. Current Snapshot: MvccSnapshot[applied={T|T < 7333088406966951936}]
I20260925 02:53:06.868719 18332 mvcc.cc:205] Tried to move back new op lower bound from 7333088406966951936 to 7333088406856437760. Current Snapshot: MvccSnapshot[applied={T|T < 7333088406966951936}]
I20260925 02:53:06.873816 17846 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:06.873950 17846 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:06.876125 17846 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14d05fc16a174d7d8793be01af6c7280: Sending DeleteTablet for 3 replicas of tablet 55d36829b0c3473689fcba7943aaace0
I20260925 02:53:06.876801 18235 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d36829b0c3473689fcba7943aaace0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:06 UTC) from {username='slave'} at 127.0.0.1:55648
I20260925 02:53:06.876799 18104 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d36829b0c3473689fcba7943aaace0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:06 UTC) from {username='slave'} at 127.0.0.1:55894
I20260925 02:53:06.877221 18340 tablet_replica.cc:330] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452: stopping tablet replica
I20260925 02:53:06.877234 17973 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d36829b0c3473689fcba7943aaace0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:06 UTC) from {username='slave'} at 127.0.0.1:41600
I20260925 02:53:06.877501 18340 raft_consensus.cc:2229] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:53:06.877621 18342 tablet_replica.cc:330] T 55d36829b0c3473689fcba7943aaace0 P 8e712a1c17644dfc97c4397ec3e10da0: stopping tablet replica
I20260925 02:53:06.877792 18340 raft_consensus.cc:2258] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:06.877873 18342 raft_consensus.cc:2229] T 55d36829b0c3473689fcba7943aaace0 P 8e712a1c17644dfc97c4397ec3e10da0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:06.878050 18341 tablet_replica.cc:330] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b: stopping tablet replica
I20260925 02:53:06.878171 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 17910
I20260925 02:53:06.878158 18342 raft_consensus.cc:2258] T 55d36829b0c3473689fcba7943aaace0 P 8e712a1c17644dfc97c4397ec3e10da0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:06.878301 18341 raft_consensus.cc:2229] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:06.878736 18341 raft_consensus.cc:2258] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:06.881748 18340 ts_tablet_manager.cc:1905] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:06.881757 18341 ts_tablet_manager.cc:1905] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:06.884383 18341 ts_tablet_manager.cc:1918] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:06.884542 18341 log.cc:1196] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/55d36829b0c3473689fcba7943aaace0
I20260925 02:53:06.884927 18341 ts_tablet_manager.cc:1939] T 55d36829b0c3473689fcba7943aaace0 P fa60c1cf54ed4395a7dfeee04a18c81b: Deleting consensus metadata
I20260925 02:53:06.885102 18340 ts_tablet_manager.cc:1918] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:06.885205 18340 log.cc:1196] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/55d36829b0c3473689fcba7943aaace0
I20260925 02:53:06.885519 18340 ts_tablet_manager.cc:1939] T 55d36829b0c3473689fcba7943aaace0 P adaa572ae8ca4c70bde845b74d200452: Deleting consensus metadata
W20260925 02:53:06.885967 17835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.222.193:45049 (error 108)
I20260925 02:53:06.886241 17834 catalog_manager.cc:4841] TS fa60c1cf54ed4395a7dfeee04a18c81b (127.12.222.195:36305): tablet 55d36829b0c3473689fcba7943aaace0 (table test-table [id=8c2d3f4409cd4da9bd534c02a6814621]) successfully deleted
I20260925 02:53:06.886525 17836 catalog_manager.cc:4841] TS adaa572ae8ca4c70bde845b74d200452 (127.12.222.194:36961): tablet 55d36829b0c3473689fcba7943aaace0 (table test-table [id=8c2d3f4409cd4da9bd534c02a6814621]) successfully deleted
I20260925 02:53:06.886970 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 18042
I20260925 02:53:06.887136 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:06.886859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:45049 (local address 127.0.0.1:41616)
0925 02:53:06.886957 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:06.887049 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.193:45049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 02:53:06.888402 17835 catalog_manager.cc:4584] TS 8e712a1c17644dfc97c4397ec3e10da0 (127.12.222.193:45049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55d36829b0c3473689fcba7943aaace0: Network error: Client connection negotiation failed: client connection to 127.12.222.193:45049: connect: Connection refused (error 111)
I20260925 02:53:06.892426 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 18173
I20260925 02:53:06.897522 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 17818
2026-09-25T02:53:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1073 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 02:53:06.918462 13179 mini_oidc.cc:152] Starting JWKS server
I20260925 02:53:06.919922 13179 webserver.cc:460] Webserver started at https://127.0.0.1:46097/ using document root <none> and password file <none>
I20260925 02:53:06.920089 13179 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:53:06.920396 13179 webserver.cc:460] Webserver started at http://127.0.0.1:43241/ using document root <none> and password file <none>
2026-09-25T02:53:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:06Z Disabled control of system clock
I20260925 02:53:06.928527 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:37549
--webserver_interface=127.12.222.254
--webserver_port=0
--builtin_ntp_servers=127.12.222.212:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:37549
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46097/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/testchainca.cert with env {}
W20260925 02:53:07.028982 18349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:07.029181 18349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:07.029215 18349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:07.032728 18349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:07.032768 18349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:07.032783 18349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:07.032797 18349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:07.032810 18349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:53:07.032824 18349 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:53:07.032836 18349 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46097/jwks.json
I20260925 02:53:07.037410 18349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:37549
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:37549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46097/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:07.038203 18349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:07.039021 18349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:07.047161 18355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:07.047128 18354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:07.047128 18357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:07.047434 18349 server_base.cc:1034] running on GCE node
I20260925 02:53:07.047986 18349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:07.049093 18349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:07.050275 18349 hybrid_clock.cc:648] HybridClock initialized: now 1790304787050254 us; error 49 us; skew 500 ppm
I20260925 02:53:07.051853 18349 webserver.cc:460] Webserver started at http://127.12.222.254:41257/ using document root <none> and password file <none>
I20260925 02:53:07.052462 18349 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:07.052551 18349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:07.052814 18349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:07.054759 18349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "f72656903d3142aba8f92a4ef9efbefa"
format_stamp: "Formatted at 2026-09-25 02:53:07 on dist-test-slave-252r"
server_key: "6aeb4b0841993656a3ac16460d17d87e"
server_key_iv: "474b08ee42c120368a31b99316f56d7d"
server_key_version: "encryptionkey@0"
I20260925 02:53:07.055284 18349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "f72656903d3142aba8f92a4ef9efbefa"
format_stamp: "Formatted at 2026-09-25 02:53:07 on dist-test-slave-252r"
server_key: "6aeb4b0841993656a3ac16460d17d87e"
server_key_iv: "474b08ee42c120368a31b99316f56d7d"
server_key_version: "encryptionkey@0"
I20260925 02:53:07.058511 18349 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:53:07.060626 18363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:07.061723 18349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:07.061864 18349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "f72656903d3142aba8f92a4ef9efbefa"
format_stamp: "Formatted at 2026-09-25 02:53:07 on dist-test-slave-252r"
server_key: "6aeb4b0841993656a3ac16460d17d87e"
server_key_iv: "474b08ee42c120368a31b99316f56d7d"
server_key_version: "encryptionkey@0"
I20260925 02:53:07.061971 18349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:07.078835 18349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:07.079384 18349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:07.079566 18349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:07.086114 18349 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:37549
I20260925 02:53:07.086130 18415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:37549 every 8 connection(s)
I20260925 02:53:07.086903 18349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:07.089658 18416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:07.093346 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 18349
I20260925 02:53:07.093436 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:53:07.093698 13179 external_mini_cluster.cc:1428] Setting key 40c161226bb31c7c89863c6c273df254
I20260925 02:53:07.094878 18416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa: Bootstrap starting.
I20260925 02:53:07.097290 18416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:07.098054 18416 log.cc:824] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:07.099737 18416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa: No bootstrap required, opened a new log
I20260925 02:53:07.103071 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:07.095030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.12.222.254:37549)
0925 02:53:07.095639 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:07.095660 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:07.095699 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:53:07.096503 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:07.096524 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:07.096954 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:07.097270 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:07.097686 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.098832 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.099658 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.100310 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.100444 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:07.101206 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:07.101214 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:07.101229 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:07.101691 (+ 462us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:07.101758 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:07.101783 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:07.101880 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:07.101982 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 02:53:07.102247 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":82,"threads_started":1}
I20260925 02:53:07.103859 18416 raft_consensus.cc:348] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72656903d3142aba8f92a4ef9efbefa" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37549 } }
I20260925 02:53:07.104045 18416 raft_consensus.cc:374] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:07.104104 18416 raft_consensus.cc:729] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f72656903d3142aba8f92a4ef9efbefa, State: Initialized, Role: FOLLOWER
I20260925 02:53:07.104504 18416 consensus_queue.cc:260] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72656903d3142aba8f92a4ef9efbefa" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37549 } }
I20260925 02:53:07.104655 18416 raft_consensus.cc:388] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:07.104746 18416 raft_consensus.cc:482] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:07.104851 18416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:07.105655 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:53:07.105697 18416 raft_consensus.cc:504] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72656903d3142aba8f92a4ef9efbefa" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37549 } }
I20260925 02:53:07.106006 18416 leader_election.cc:302] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f72656903d3142aba8f92a4ef9efbefa; no voters:
I20260925 02:53:07.106292 18416 leader_election.cc:288] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:53:07.106357 18421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:07.106580 18421 raft_consensus.cc:686] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa [term 1 LEADER]: Becoming Leader. State: Replica: f72656903d3142aba8f92a4ef9efbefa, State: Running, Role: LEADER
I20260925 02:53:07.106851 18421 consensus_queue.cc:237] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72656903d3142aba8f92a4ef9efbefa" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37549 } }
I20260925 02:53:07.107434 18416 sys_catalog.cc:564] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:07.107933 18423 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f72656903d3142aba8f92a4ef9efbefa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72656903d3142aba8f92a4ef9efbefa" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37549 } } }
I20260925 02:53:07.108053 18423 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:07.108206 18422 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f72656903d3142aba8f92a4ef9efbefa. Latest consensus state: current_term: 1 leader_uuid: "f72656903d3142aba8f92a4ef9efbefa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72656903d3142aba8f92a4ef9efbefa" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37549 } } }
I20260925 02:53:07.108362 18422 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:07.109488 18426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:07.112489 18426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:07.117329 18426 catalog_manager.cc:1347] Generated new cluster ID: cc67629e7b9d4bafbd03d72bce78b3ff
I20260925 02:53:07.117383 18426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:07.144016 18426 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:07.144910 18426 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:07.151082 18426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f72656903d3142aba8f92a4ef9efbefa: Generated new TSK 0
I20260925 02:53:07.151742 18426 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 02:53:07.178540 18447 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 02:53:07.178694 18447 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 02:53:07.179131 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:07.163600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36856 (local address 127.12.222.254:37549)
0925 02:53:07.163729 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:07.163733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:07.163988 (+ 255us) server_negotiation.cc:407] Connection header received
0925 02:53:07.164204 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:07.164206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:07.164310 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:07.164383 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:53:07.164662 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.165219 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.166348 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.166554 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.166628 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:07.166991 (+ 363us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:53:07.178791 (+ 11800us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 02:53:07.179026 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36856: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 02:53:07.179216 18419 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36856: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 02:53:07.180495 18445 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:07.163559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37549 (local address 127.0.0.1:36856)
0925 02:53:07.163902 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:07.163917 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:07.164047 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:07.164421 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:07.164424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:07.164432 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:53:07.164554 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:07.164560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.165316 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.165320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:07.166251 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:07.166260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.166659 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.166662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:07.166755 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:07.166809 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:53:07.179027 (+ 12218us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 02:53:07.179226 (+ 199us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.222.254:37549: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":113,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.12.222.254:37549: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 02:53:07.181245 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 18349
2026-09-25T02:53:07Z chronyd exiting
I20260925 02:53:07.218942 13179 test_util.cc:175] -----------------------------------------------
I20260925 02:53:07.219000 13179 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304762911399-13179-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (302 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 02:53:07.219843 13179 mini_oidc.cc:152] Starting JWKS server
I20260925 02:53:07.220630 13179 webserver.cc:460] Webserver started at https://127.0.0.1:33739/ using document root <none> and password file <none>
I20260925 02:53:07.220703 13179 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:53:07.220925 13179 webserver.cc:460] Webserver started at http://127.0.0.1:38951/ using document root <none> and password file <none>
2026-09-25T02:53:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:07Z Disabled control of system clock
I20260925 02:53:07.229372 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:39119
--webserver_interface=127.12.222.254
--webserver_port=0
--builtin_ntp_servers=127.12.222.212:46541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:39119
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33739/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/testchainca.cert with env {}
W20260925 02:53:07.331398 18453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:07.331600 18453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:07.331651 18453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:07.335561 18453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:07.335608 18453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:07.335623 18453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:07.335638 18453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:07.335652 18453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:53:07.335667 18453 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:53:07.335680 18453 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33739/jwks.json
I20260925 02:53:07.340385 18453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:39119
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:39119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33739/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:07.341248 18453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:07.342139 18453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:07.350075 18458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:07.350075 18459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:07.350206 18453 server_base.cc:1034] running on GCE node
W20260925 02:53:07.350373 18461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:07.350935 18453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:07.351918 18453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:07.353091 18453 hybrid_clock.cc:648] HybridClock initialized: now 1790304787353050 us; error 52 us; skew 500 ppm
I20260925 02:53:07.354712 18453 webserver.cc:460] Webserver started at http://127.12.222.254:40055/ using document root <none> and password file <none>
I20260925 02:53:07.355229 18453 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:07.355309 18453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:07.355532 18453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:07.357394 18453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "bcba864a5ae94059a83fcf0d0342be8e"
format_stamp: "Formatted at 2026-09-25 02:53:07 on dist-test-slave-252r"
server_key: "eec41b1511fd155e7d108135432bfe48"
server_key_iv: "2f0ee5e1f7613126a861f9855b0b5005"
server_key_version: "encryptionkey@0"
I20260925 02:53:07.357944 18453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "bcba864a5ae94059a83fcf0d0342be8e"
format_stamp: "Formatted at 2026-09-25 02:53:07 on dist-test-slave-252r"
server_key: "eec41b1511fd155e7d108135432bfe48"
server_key_iv: "2f0ee5e1f7613126a861f9855b0b5005"
server_key_version: "encryptionkey@0"
I20260925 02:53:07.361156 18453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:53:07.363176 18467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:07.364307 18453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 02:53:07.364429 18453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "bcba864a5ae94059a83fcf0d0342be8e"
format_stamp: "Formatted at 2026-09-25 02:53:07 on dist-test-slave-252r"
server_key: "eec41b1511fd155e7d108135432bfe48"
server_key_iv: "2f0ee5e1f7613126a861f9855b0b5005"
server_key_version: "encryptionkey@0"
I20260925 02:53:07.364534 18453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:07.374877 18453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:07.375422 18453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:07.375602 18453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:07.381531 18453 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:39119
I20260925 02:53:07.381536 18519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:39119 every 8 connection(s)
I20260925 02:53:07.382321 18453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:07.383806 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 18453
I20260925 02:53:07.383958 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:53:07.384294 13179 external_mini_cluster.cc:1428] Setting key c4ee313f3bd73f74573aab1f6901d462
I20260925 02:53:07.385725 18520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:07.391232 18520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e: Bootstrap starting.
I20260925 02:53:07.392534 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:07.385468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51924 (local address 127.12.222.254:39119)
0925 02:53:07.386024 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:07.386038 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:07.386068 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:53:07.386624 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:07.386638 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:07.386960 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:07.387208 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:07.387530 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.388499 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.389630 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.390046 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.390150 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:07.390925 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:07.390929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:07.390939 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:07.391256 (+ 317us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:07.391305 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:07.391322 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:07.391432 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:07.391571 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 02:53:07.391816 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":127,"threads_started":1}
I20260925 02:53:07.393378 18520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:07.394073 18520 log.cc:824] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:07.395015 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:53:07.396023 18520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e: No bootstrap required, opened a new log
I20260925 02:53:07.399039 18520 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcba864a5ae94059a83fcf0d0342be8e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39119 } }
I20260925 02:53:07.399251 18520 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:07.399328 18520 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcba864a5ae94059a83fcf0d0342be8e, State: Initialized, Role: FOLLOWER
I20260925 02:53:07.399706 18520 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcba864a5ae94059a83fcf0d0342be8e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39119 } }
I20260925 02:53:07.399842 18520 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:07.399921 18520 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:07.400022 18520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:07.400827 18520 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcba864a5ae94059a83fcf0d0342be8e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39119 } }
I20260925 02:53:07.401163 18520 leader_election.cc:302] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcba864a5ae94059a83fcf0d0342be8e; no voters:
I20260925 02:53:07.401435 18520 leader_election.cc:288] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:53:07.401540 18525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:07.401786 18525 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e [term 1 LEADER]: Becoming Leader. State: Replica: bcba864a5ae94059a83fcf0d0342be8e, State: Running, Role: LEADER
I20260925 02:53:07.402104 18525 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcba864a5ae94059a83fcf0d0342be8e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39119 } }
I20260925 02:53:07.402649 18520 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:07.403106 18527 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcba864a5ae94059a83fcf0d0342be8e. Latest consensus state: current_term: 1 leader_uuid: "bcba864a5ae94059a83fcf0d0342be8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcba864a5ae94059a83fcf0d0342be8e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39119 } } }
I20260925 02:53:07.403227 18527 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:07.403499 18526 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcba864a5ae94059a83fcf0d0342be8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcba864a5ae94059a83fcf0d0342be8e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39119 } } }
I20260925 02:53:07.403590 18526 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:07.403650 18530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:07.406211 18530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:07.410771 18530 catalog_manager.cc:1347] Generated new cluster ID: 5b70585b6a6f4564a9350507685c9545
I20260925 02:53:07.410830 18530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:07.418184 18530 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:07.418887 18530 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:07.424538 18530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcba864a5ae94059a83fcf0d0342be8e: Generated new TSK 0
I20260925 02:53:07.425078 18530 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 02:53:07.445598 18550 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 02:53:07.445673 18550 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 02:53:07.446210 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:07.431737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51932 (local address 127.12.222.254:39119)
0925 02:53:07.431841 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:07.431845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:07.431992 (+ 147us) server_negotiation.cc:407] Connection header received
0925 02:53:07.432097 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:07.432100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:07.432145 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:07.432257 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:53:07.432462 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.433157 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.433990 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.434332 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.434435 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:07.434697 (+ 262us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:53:07.445872 (+ 11175us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 02:53:07.446080 (+ 208us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51932: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 02:53:07.446229 18549 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:07.431641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:39119 (local address 127.0.0.1:51932)
0925 02:53:07.431924 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:07.431934 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:07.432013 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:07.432282 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:07.432284 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:07.432289 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:53:07.432371 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:07.432375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.433288 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.433290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:07.433888 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:07.433894 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.434454 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.434456 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:07.434530 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:07.434550 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:53:07.446011 (+ 11461us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 02:53:07.446112 (+ 101us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.222.254:39119: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":84,"threads_started":1}
W20260925 02:53:07.446270 18523 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51932: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260925 02:53:07.446827 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 18453
2026-09-25T02:53:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (265 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 02:53:07.485960 13179 mini_oidc.cc:152] Starting JWKS server
I20260925 02:53:07.486967 13179 webserver.cc:460] Webserver started at https://127.0.0.1:41467/ using document root <none> and password file <none>
I20260925 02:53:07.487089 13179 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:53:07.487351 13179 webserver.cc:460] Webserver started at http://127.0.0.1:38755/ using document root <none> and password file <none>
2026-09-25T02:53:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:07Z Disabled control of system clock
I20260925 02:53:07.495601 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:36227
--webserver_interface=127.12.222.254
--webserver_port=0
--builtin_ntp_servers=127.12.222.212:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:36227
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41467/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 02:53:07.598407 18556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:07.598623 18556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:07.598657 18556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:07.602366 18556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:07.602411 18556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:07.602425 18556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:07.602439 18556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:07.602453 18556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:53:07.602468 18556 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:53:07.602481 18556 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41467/jwks.json
I20260925 02:53:07.606709 18556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:36227
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:36227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41467/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:07.607482 18556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:07.608640 18556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:07.616365 18562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:07.616372 18561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:07.616434 18556 server_base.cc:1034] running on GCE node
W20260925 02:53:07.616372 18564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:07.617147 18556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:07.618129 18556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:07.619279 18556 hybrid_clock.cc:648] HybridClock initialized: now 1790304787619258 us; error 34 us; skew 500 ppm
I20260925 02:53:07.625955 18556 webserver.cc:460] Webserver started at http://127.12.222.254:40013/ using document root <none> and password file <none>
I20260925 02:53:07.626708 18556 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:07.626806 18556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:07.627024 18556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:07.629566 18556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "47d368077a14436b86f676d9f6f604d2"
format_stamp: "Formatted at 2026-09-25 02:53:07 on dist-test-slave-252r"
server_key: "b5b475c1fba376ee31ae6163f75577fe"
server_key_iv: "80ae26c304acedb10ebe7bc373e90ab1"
server_key_version: "encryptionkey@0"
I20260925 02:53:07.630283 18556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "47d368077a14436b86f676d9f6f604d2"
format_stamp: "Formatted at 2026-09-25 02:53:07 on dist-test-slave-252r"
server_key: "b5b475c1fba376ee31ae6163f75577fe"
server_key_iv: "80ae26c304acedb10ebe7bc373e90ab1"
server_key_version: "encryptionkey@0"
I20260925 02:53:07.634896 18556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:53:07.638026 18570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:07.639716 18556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:53:07.639844 18556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "47d368077a14436b86f676d9f6f604d2"
format_stamp: "Formatted at 2026-09-25 02:53:07 on dist-test-slave-252r"
server_key: "b5b475c1fba376ee31ae6163f75577fe"
server_key_iv: "80ae26c304acedb10ebe7bc373e90ab1"
server_key_version: "encryptionkey@0"
I20260925 02:53:07.639955 18556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:07.663699 18556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:07.664538 18556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:07.664734 18556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:07.673532 18556 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:36227
I20260925 02:53:07.673542 18623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:36227 every 8 connection(s)
I20260925 02:53:07.674500 18556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:07.677387 18624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:07.680560 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 18556
I20260925 02:53:07.680671 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:53:07.680953 13179 external_mini_cluster.cc:1428] Setting key 9f9e5febd1895cc41b844b49dd7f5dd4
I20260925 02:53:07.682999 18624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2: Bootstrap starting.
I20260925 02:53:07.685286 18624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:07.686168 18624 log.cc:824] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:07.688282 18624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2: No bootstrap required, opened a new log
I20260925 02:53:07.689832 18627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:07.682144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40522 (local address 127.12.222.254:36227)
0925 02:53:07.682670 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:07.682685 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:07.682717 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:53:07.683375 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:07.683392 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:07.683763 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:07.684058 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:07.684449 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.685431 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.686584 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.687161 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.687339 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:07.688159 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:07.688164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:07.688175 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:07.688626 (+ 451us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:07.688677 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:07.688696 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:07.688800 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:07.688953 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 02:53:07.689201 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":107,"threads_started":1}
I20260925 02:53:07.691677 18624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d368077a14436b86f676d9f6f604d2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 36227 } }
I20260925 02:53:07.691955 18624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:07.692042 18624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d368077a14436b86f676d9f6f604d2, State: Initialized, Role: FOLLOWER
I20260925 02:53:07.692512 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:53:07.692509 18624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d368077a14436b86f676d9f6f604d2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 36227 } }
I20260925 02:53:07.692662 18624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:07.692719 18624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:07.692803 18624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:07.693797 18624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d368077a14436b86f676d9f6f604d2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 36227 } }
I20260925 02:53:07.694154 18624 leader_election.cc:302] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47d368077a14436b86f676d9f6f604d2; no voters:
I20260925 02:53:07.694475 18624 leader_election.cc:288] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:53:07.694602 18629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:07.694854 18629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2 [term 1 LEADER]: Becoming Leader. State: Replica: 47d368077a14436b86f676d9f6f604d2, State: Running, Role: LEADER
I20260925 02:53:07.695145 18629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d368077a14436b86f676d9f6f604d2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 36227 } }
I20260925 02:53:07.695601 18624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:07.696193 18629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47d368077a14436b86f676d9f6f604d2. Latest consensus state: current_term: 1 leader_uuid: "47d368077a14436b86f676d9f6f604d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d368077a14436b86f676d9f6f604d2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 36227 } } }
I20260925 02:53:07.696334 18629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:07.696686 18631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47d368077a14436b86f676d9f6f604d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d368077a14436b86f676d9f6f604d2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 36227 } } }
I20260925 02:53:07.696775 18631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:07.697009 18637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:07.699612 18637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:07.704111 18637 catalog_manager.cc:1347] Generated new cluster ID: 65a982bb1bb94a9b8ff66814cd3787c9
I20260925 02:53:07.704171 18637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:07.711251 18637 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:07.712314 18637 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:07.720467 18637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47d368077a14436b86f676d9f6f604d2: Generated new TSK 0
I20260925 02:53:07.721206 18637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:53:07.738181 18627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:07.733439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.12.222.254:36227)
0925 02:53:07.733560 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:07.733564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:07.733754 (+ 190us) server_negotiation.cc:407] Connection header received
0925 02:53:07.733934 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:07.733936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:07.733984 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:07.734065 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:53:07.734352 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.734838 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.736049 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.736295 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.736367 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:07.736678 (+ 311us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:53:07.737808 (+ 1130us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41467/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 02:53:07.738034 (+ 226us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40526: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41467/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 02:53:07.738245 18627 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40526: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41467/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 02:53:07.740031 18653 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:07.733372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:36227 (local address 127.0.0.1:40526)
0925 02:53:07.733668 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:07.733681 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:07.733787 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:07.734093 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:07.734096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:07.734105 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:53:07.734220 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:07.734227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.734961 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.734966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:07.735939 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:07.735949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.736399 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.736403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:07.736499 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:07.736527 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:53:07.737972 (+ 1445us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41467/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 02:53:07.739860 (+ 1888us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.222.254:36227: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41467/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":1720,"thread_start_us":118,"threads_started":1}
I20260925 02:53:07.740557 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 18556
E20260925 02:53:07.748608 18654 webserver.cc:533] Webserver: sslize failed:
E20260925 02:53:07.748682 18654 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T02:53:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (293 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T02:53:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:07Z Disabled control of system clock
I20260925 02:53:07.786900 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:39795
--webserver_interface=127.12.222.254
--webserver_port=0
--builtin_ntp_servers=127.12.222.212:35785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:39795
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/testchainca.cert with env {}
W20260925 02:53:07.889324 18658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:07.889585 18658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:07.889672 18658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:07.893427 18658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:07.893497 18658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:07.893524 18658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:07.893558 18658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:07.893589 18658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:53:07.893620 18658 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:53:07.893647 18658 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 02:53:07.898561 18658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:35785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:39795
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:39795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:07.899464 18658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:07.900384 18658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:07.908368 18666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:07.908355 18663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:07.908411 18658 server_base.cc:1034] running on GCE node
W20260925 02:53:07.908368 18664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:07.909004 18658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:07.909909 18658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:07.911090 18658 hybrid_clock.cc:648] HybridClock initialized: now 1790304787911066 us; error 36 us; skew 500 ppm
I20260925 02:53:07.912716 18658 webserver.cc:460] Webserver started at http://127.12.222.254:36981/ using document root <none> and password file <none>
I20260925 02:53:07.913260 18658 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:07.913350 18658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:07.913559 18658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:07.915402 18658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "e54b7e4090c344dcafad3b688de42e00"
format_stamp: "Formatted at 2026-09-25 02:53:07 on dist-test-slave-252r"
server_key: "b03c9d11e2284f1eca9d4ab1048f3e41"
server_key_iv: "dfeac6dd7e9819101ce7f166e6781aac"
server_key_version: "encryptionkey@0"
I20260925 02:53:07.915931 18658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "e54b7e4090c344dcafad3b688de42e00"
format_stamp: "Formatted at 2026-09-25 02:53:07 on dist-test-slave-252r"
server_key: "b03c9d11e2284f1eca9d4ab1048f3e41"
server_key_iv: "dfeac6dd7e9819101ce7f166e6781aac"
server_key_version: "encryptionkey@0"
I20260925 02:53:07.919275 18658 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 02:53:07.921557 18672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:07.922643 18658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:53:07.922768 18658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "e54b7e4090c344dcafad3b688de42e00"
format_stamp: "Formatted at 2026-09-25 02:53:07 on dist-test-slave-252r"
server_key: "b03c9d11e2284f1eca9d4ab1048f3e41"
server_key_iv: "dfeac6dd7e9819101ce7f166e6781aac"
server_key_version: "encryptionkey@0"
I20260925 02:53:07.922865 18658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:07.945219 18658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:07.945827 18658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:07.946018 18658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:07.952435 18658 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:39795
I20260925 02:53:07.952450 18724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:39795 every 8 connection(s)
I20260925 02:53:07.953378 18658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:07.956035 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:07.961146 18725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00: Bootstrap starting.
I20260925 02:53:07.962119 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 18658
I20260925 02:53:07.962262 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:53:07.962566 13179 external_mini_cluster.cc:1428] Setting key 9a16b73bc8026534e0b7609b2ea5146b
I20260925 02:53:07.963963 18725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:07.964731 18725 log.cc:824] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:07.966498 18725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00: No bootstrap required, opened a new log
I20260925 02:53:07.969921 18725 raft_consensus.cc:348] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54b7e4090c344dcafad3b688de42e00" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39795 } }
I20260925 02:53:07.970158 18725 raft_consensus.cc:374] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:07.970225 18725 raft_consensus.cc:729] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e54b7e4090c344dcafad3b688de42e00, State: Initialized, Role: FOLLOWER
I20260925 02:53:07.970661 18725 consensus_queue.cc:260] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54b7e4090c344dcafad3b688de42e00" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39795 } }
I20260925 02:53:07.970806 18725 raft_consensus.cc:388] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:07.970901 18725 raft_consensus.cc:482] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:07.971021 18725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:07.971208 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:07.963586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51330 (local address 127.12.222.254:39795)
0925 02:53:07.964263 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:07.964277 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:07.964311 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:53:07.964951 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:07.964968 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:07.965346 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:07.965670 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:07.966075 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.967047 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.967900 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:07.968426 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:07.968596 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:07.969461 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:07.969466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:07.969478 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:07.969821 (+ 343us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:07.969874 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:07.969893 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:07.970007 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:07.970149 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 02:53:07.970411 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":115,"threads_started":1}
I20260925 02:53:07.972080 18725 raft_consensus.cc:504] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54b7e4090c344dcafad3b688de42e00" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39795 } }
I20260925 02:53:07.972424 18725 leader_election.cc:302] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e54b7e4090c344dcafad3b688de42e00; no voters:
I20260925 02:53:07.972751 18725 leader_election.cc:288] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:53:07.972816 18730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:07.973027 18730 raft_consensus.cc:686] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00 [term 1 LEADER]: Becoming Leader. State: Replica: e54b7e4090c344dcafad3b688de42e00, State: Running, Role: LEADER
I20260925 02:53:07.973299 18730 consensus_queue.cc:237] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54b7e4090c344dcafad3b688de42e00" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39795 } }
I20260925 02:53:07.973989 18725 sys_catalog.cc:564] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:07.974167 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:53:07.974818 18731 sys_catalog.cc:455] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e54b7e4090c344dcafad3b688de42e00. Latest consensus state: current_term: 1 leader_uuid: "e54b7e4090c344dcafad3b688de42e00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54b7e4090c344dcafad3b688de42e00" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39795 } } }
I20260925 02:53:07.975207 18731 sys_catalog.cc:458] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:07.975098 18732 sys_catalog.cc:455] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e54b7e4090c344dcafad3b688de42e00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54b7e4090c344dcafad3b688de42e00" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39795 } } }
I20260925 02:53:07.975488 18732 sys_catalog.cc:458] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:07.975692 18739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:07.978416 18739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:07.983150 18739 catalog_manager.cc:1347] Generated new cluster ID: f3ec7cbcbf324b6a8001cc2da94c120b
I20260925 02:53:07.983209 18739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:08.003206 18739 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:08.004319 18739 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:08.013108 18739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e54b7e4090c344dcafad3b688de42e00: Generated new TSK 0
I20260925 02:53:08.013777 18739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:53:08.495966 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:08.025377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51334 (local address 127.12.222.254:39795)
0925 02:53:08.025516 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:08.025521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:08.025718 (+ 197us) server_negotiation.cc:407] Connection header received
0925 02:53:08.025847 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:08.025849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:08.025892 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:08.025974 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:53:08.026238 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:08.026838 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:08.028033 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:08.028253 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:08.028364 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:08.028717 (+ 353us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:53:08.495582 (+466865us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 02:53:08.495846 (+ 264us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51334: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 02:53:08.496071 18728 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51334: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 02:53:08.496526 18754 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:08.025317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:39795 (local address 127.0.0.1:51334)
0925 02:53:08.025627 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:08.025641 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:08.025748 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:08.025999 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:08.026002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:08.026010 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:53:08.026138 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:08.026144 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:08.026960 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:08.026964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:08.027927 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:08.027936 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:08.028398 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:08.028402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:08.028509 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:08.028556 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:53:08.495811 (+467255us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 02:53:08.496007 (+ 196us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.222.254:39795: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":119,"threads_started":1}
I20260925 02:53:08.497131 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 18658
2026-09-25T02:53:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (737 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (118 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (117 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:53:09 dist-test-slave-252r krb5kdc[18766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:53:09 dist-test-slave-252r krb5kdc[18766](info): set up 2 sockets
Sep 25 02:53:09 dist-test-slave-252r krb5kdc[18766](info): commencing operation
krb5kdc: starting...
W20260925 02:53:10.892055 13179 mini_kdc.cc:121] Time spent starting KDC: real 2.138s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:53:10 dist-test-slave-252r krb5kdc[18766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:53:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:53:11.001134 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:37857
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:35211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:37857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:11.102198 18779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:11.102454 18779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:11.102516 18779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:11.106283 18779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:11.106350 18779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:11.106382 18779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:11.106415 18779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:11.106451 18779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:11.111105 18779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:35211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:37857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:37857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:11.111979 18779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:11.112896 18779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:11.120287 18787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:11.120328 18784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:11.120328 18785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:11.120435 18779 server_base.cc:1034] running on GCE node
I20260925 02:53:11.121057 18779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:11.121996 18779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:11.123195 18779 hybrid_clock.cc:648] HybridClock initialized: now 1790304791123202 us; error 70 us; skew 500 ppm
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[18766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304791, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:11.129496 18779 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:53:11.130294 18779 webserver.cc:460] Webserver started at http://127.12.222.254:42423/ using document root <none> and password file <none>
I20260925 02:53:11.130776 18779 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:11.130860 18779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:11.131069 18779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:11.132959 18779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "5c485e01e51b4c248942f9cc148bc601"
format_stamp: "Formatted at 2026-09-25 02:53:11 on dist-test-slave-252r"
server_key: "6a80c40c6ec30082bdb52c7d7b631e01"
server_key_iv: "a7bbc271aac07a00e8214c9c98d57462"
server_key_version: "encryptionkey@0"
I20260925 02:53:11.133504 18779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "5c485e01e51b4c248942f9cc148bc601"
format_stamp: "Formatted at 2026-09-25 02:53:11 on dist-test-slave-252r"
server_key: "6a80c40c6ec30082bdb52c7d7b631e01"
server_key_iv: "a7bbc271aac07a00e8214c9c98d57462"
server_key_version: "encryptionkey@0"
I20260925 02:53:11.136869 18779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:53:11.138962 18794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:11.139995 18779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:11.140125 18779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "5c485e01e51b4c248942f9cc148bc601"
format_stamp: "Formatted at 2026-09-25 02:53:11 on dist-test-slave-252r"
server_key: "6a80c40c6ec30082bdb52c7d7b631e01"
server_key_iv: "a7bbc271aac07a00e8214c9c98d57462"
server_key_version: "encryptionkey@0"
I20260925 02:53:11.140223 18779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:11.164673 18779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:11.167636 18779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:11.167793 18779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:11.175083 18779 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:37857
I20260925 02:53:11.175108 18846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:37857 every 8 connection(s)
I20260925 02:53:11.175912 18779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:11.176119 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 18779
I20260925 02:53:11.176246 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:53:11.176529 13179 external_mini_cluster.cc:1428] Setting key 40aaee2644e92aa8979f06575149342b
I20260925 02:53:11.178936 18847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:11.185108 18847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601: Bootstrap starting.
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[18766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:11.187690 18847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:11.188555 18847 log.cc:824] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:11.190362 18847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601: No bootstrap required, opened a new log
I20260925 02:53:11.190631 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:11.177821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41728 (local address 127.12.222.254:37857)
0925 02:53:11.178207 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:11.178224 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:11.178262 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:53:11.179034 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:11.179056 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:11.179420 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:11.179745 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:11.180487 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.181957 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.182753 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.183410 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.183569 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:11.186008 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:11.186014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:11.186028 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:11.186066 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:11.187858 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:11.188289 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:11.188294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:11.188300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:11.188447 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:11.188727 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:11.188735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:11.188736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:11.189214 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:11.189400 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:11.189696 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 02:53:11.189948 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
I20260925 02:53:11.194041 18847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c485e01e51b4c248942f9cc148bc601" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37857 } }
I20260925 02:53:11.194281 18847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:11.194361 18847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c485e01e51b4c248942f9cc148bc601, State: Initialized, Role: FOLLOWER
I20260925 02:53:11.194756 18847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c485e01e51b4c248942f9cc148bc601" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37857 } }
I20260925 02:53:11.194900 18847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:11.194986 18847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:11.195103 18847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:11.195917 18847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c485e01e51b4c248942f9cc148bc601" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37857 } }
I20260925 02:53:11.196331 18847 leader_election.cc:302] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c485e01e51b4c248942f9cc148bc601; no voters:
I20260925 02:53:11.196617 18847 leader_election.cc:288] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:53:11.196702 18852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:11.196973 18852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601 [term 1 LEADER]: Becoming Leader. State: Replica: 5c485e01e51b4c248942f9cc148bc601, State: Running, Role: LEADER
I20260925 02:53:11.197300 18852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c485e01e51b4c248942f9cc148bc601" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37857 } }
I20260925 02:53:11.197773 18847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:11.198827 18853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c485e01e51b4c248942f9cc148bc601" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c485e01e51b4c248942f9cc148bc601" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37857 } } }
I20260925 02:53:11.198944 18853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:11.199235 18854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c485e01e51b4c248942f9cc148bc601. Latest consensus state: current_term: 1 leader_uuid: "5c485e01e51b4c248942f9cc148bc601" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c485e01e51b4c248942f9cc148bc601" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37857 } } }
I20260925 02:53:11.199322 18854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:11.199409 18861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:11.201969 18861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:11.206888 18861 catalog_manager.cc:1347] Generated new cluster ID: cd836464bb034b08b289fe4671e721ec
I20260925 02:53:11.206950 18861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:11.229665 18861 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:11.230438 18861 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:11.234221 18861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c485e01e51b4c248942f9cc148bc601: Generated new TSK 0
I20260925 02:53:11.234838 18861 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
I20260925 02:53:11.267400 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:37857
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:35211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:11.372872 18873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:11.373121 18873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:11.373190 18873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:11.376907 18873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:11.376981 18873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:11.377060 18873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:53:11.381734 18873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:35211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:11.382664 18873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:11.383596 18873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:11.392483 18878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:11.392489 18879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:11.392634 18873 server_base.cc:1034] running on GCE node
W20260925 02:53:11.392834 18881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:11.393178 18873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:11.393761 18873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:11.394934 18873 hybrid_clock.cc:648] HybridClock initialized: now 1790304791394906 us; error 38 us; skew 500 ppm
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[18766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304791, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:11.398015 18873 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:53:11.398736 18873 webserver.cc:460] Webserver started at http://127.12.222.193:43077/ using document root <none> and password file <none>
I20260925 02:53:11.399268 18873 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:11.399327 18873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:11.399477 18873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:11.401350 18873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "5619ea7286f84e9d9e19ffc63cdb2ee0"
format_stamp: "Formatted at 2026-09-25 02:53:11 on dist-test-slave-252r"
server_key: "f8a84edde962842260c3a31770471e4c"
server_key_iv: "a6b0c90aadbd297950a81ad7885e0f0e"
server_key_version: "encryptionkey@0"
I20260925 02:53:11.401831 18873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "5619ea7286f84e9d9e19ffc63cdb2ee0"
format_stamp: "Formatted at 2026-09-25 02:53:11 on dist-test-slave-252r"
server_key: "f8a84edde962842260c3a31770471e4c"
server_key_iv: "a6b0c90aadbd297950a81ad7885e0f0e"
server_key_version: "encryptionkey@0"
I20260925 02:53:11.405167 18873 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:53:11.407316 18888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:11.408483 18873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:53:11.408596 18873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "5619ea7286f84e9d9e19ffc63cdb2ee0"
format_stamp: "Formatted at 2026-09-25 02:53:11 on dist-test-slave-252r"
server_key: "f8a84edde962842260c3a31770471e4c"
server_key_iv: "a6b0c90aadbd297950a81ad7885e0f0e"
server_key_version: "encryptionkey@0"
I20260925 02:53:11.408684 18873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:11.423077 18873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:11.425523 18873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:11.425693 18873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:11.426194 18873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:11.426911 18873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:11.426959 18873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:11.427022 18873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:11.427053 18873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:11.435549 18873 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:46563
I20260925 02:53:11.435591 19002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:46563 every 8 connection(s)
I20260925 02:53:11.436335 18873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:53:11.443085 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 18873
I20260925 02:53:11.443202 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:53:11.443511 13179 external_mini_cluster.cc:1428] Setting key d28264f7c348ae084ae9893d5a6d3466
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[18766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304791, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:11.449718 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:11.438039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:51209 (local address 127.12.222.254:37857)
0925 02:53:11.438172 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:11.438176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:11.438921 (+ 745us) server_negotiation.cc:407] Connection header received
0925 02:53:11.440102 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:11.440104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:11.440147 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:11.440216 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:11.441327 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.441829 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.442945 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.443307 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.443415 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:11.445934 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:11.445937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:11.445940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:11.445970 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:11.447553 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:11.448123 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:11.448126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:11.448127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:11.448204 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:11.448645 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:11.448647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:11.448648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:11.448808 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:11.448885 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:11.449505 (+ 620us) server_negotiation.cc:299] Negotiation successful
0925 02:53:11.449601 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:53:11.451030 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:11.438379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37857 (local address 127.12.222.193:51209)
0925 02:53:11.438780 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:11.438818 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:11.439924 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:11.440433 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:11.440439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:11.440752 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:11.441203 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:11.441215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.441931 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.441933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:11.442737 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:11.442745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.443450 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.443453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:11.443618 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:11.444371 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:11.444394 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:11.445772 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:11.447651 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:11.447659 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:11.447675 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:11.448016 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:11.448334 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:11.448339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:11.448344 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:11.448547 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:11.448920 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:11.448929 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:11.449236 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 02:53:11.449462 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 02:53:11.449808 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":97,"threads_started":1}
I20260925 02:53:11.452193 19003 heartbeater.cc:347] Connected to a master server at 127.12.222.254:37857
I20260925 02:53:11.452456 19003 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:11.453001 19003 heartbeater.cc:511] Master 127.12.222.254:37857 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.222.194@KRBTEST.COM; defaulting to no policy
I20260925 02:53:11.454511 18811 ts_manager.cc:194] Registered new tserver with Master: 5619ea7286f84e9d9e19ffc63cdb2ee0 (127.12.222.193:46563)
I20260925 02:53:11.455807 18811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:51209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
I20260925 02:53:11.471269 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:37857
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:35211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:11.572994 19008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:11.573203 19008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:11.573242 19008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:11.577103 19008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:11.577149 19008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:11.577201 19008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:53:11.581718 19008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:35211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:11.582583 19008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:11.583407 19008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:11.592358 19014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:11.592355 19017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:11.592358 19015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:11.592553 19008 server_base.cc:1034] running on GCE node
I20260925 02:53:11.593039 19008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:11.593653 19008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:11.594838 19008 hybrid_clock.cc:648] HybridClock initialized: now 1790304791594826 us; error 35 us; skew 500 ppm
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[18766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304791, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:11.597546 19008 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:53:11.598465 19008 webserver.cc:460] Webserver started at http://127.12.222.194:39129/ using document root <none> and password file <none>
I20260925 02:53:11.599010 19008 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:11.599102 19008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:11.599320 19008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:11.601331 19008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "9ff45f8398f24c4c954879f70e0f6598"
format_stamp: "Formatted at 2026-09-25 02:53:11 on dist-test-slave-252r"
server_key: "1734de4427452cdeaae06bbc8f0ea275"
server_key_iv: "f0a8a9f64335e207eeb95781bc02268a"
server_key_version: "encryptionkey@0"
I20260925 02:53:11.601892 19008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "9ff45f8398f24c4c954879f70e0f6598"
format_stamp: "Formatted at 2026-09-25 02:53:11 on dist-test-slave-252r"
server_key: "1734de4427452cdeaae06bbc8f0ea275"
server_key_iv: "f0a8a9f64335e207eeb95781bc02268a"
server_key_version: "encryptionkey@0"
I20260925 02:53:11.605417 19008 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:53:11.607605 19024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:11.608785 19008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:53:11.608914 19008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "9ff45f8398f24c4c954879f70e0f6598"
format_stamp: "Formatted at 2026-09-25 02:53:11 on dist-test-slave-252r"
server_key: "1734de4427452cdeaae06bbc8f0ea275"
server_key_iv: "f0a8a9f64335e207eeb95781bc02268a"
server_key_version: "encryptionkey@0"
I20260925 02:53:11.609025 19008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:11.626174 19008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:11.628764 19008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:11.628970 19008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:11.629495 19008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:11.630295 19008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:11.630367 19008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:11.630446 19008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:11.630494 19008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:11.638720 19008 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:40847
I20260925 02:53:11.638736 19140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:40847 every 8 connection(s)
I20260925 02:53:11.639458 19008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:53:11.646230 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 19008
I20260925 02:53:11.646355 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:53:11.646624 13179 external_mini_cluster.cc:1428] Setting key 3d1ef46e0d6f06f480ca4196a524885f
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[18766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304791, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:11.652015 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:11.641072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:42749 (local address 127.12.222.254:37857)
0925 02:53:11.641223 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:11.641228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:11.641938 (+ 710us) server_negotiation.cc:407] Connection header received
0925 02:53:11.643107 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:11.643110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:11.643152 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:11.643218 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:11.644347 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.644874 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.645903 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.646108 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.646215 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:11.648738 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:11.648744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:11.648749 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:11.648779 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:11.650102 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:11.650570 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:11.650577 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:11.650578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:11.650666 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:11.651041 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:11.651044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:11.651045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:11.651210 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:11.651287 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:11.651816 (+ 529us) server_negotiation.cc:299] Negotiation successful
0925 02:53:11.651913 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:53:11.653115 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:11.641403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37857 (local address 127.12.222.194:42749)
0925 02:53:11.641801 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:11.641837 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:11.642903 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:11.643408 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:11.643420 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:11.643710 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:11.644175 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:11.644189 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.644979 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.644981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:11.645803 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:11.645811 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.646255 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.646259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:11.646485 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:11.647030 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:11.647045 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:11.648598 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:11.650219 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:11.650224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:11.650236 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:11.650466 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:11.650757 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:11.650763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:11.650768 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:11.650937 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:11.651315 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:11.651323 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:11.651596 (+ 273us) client_negotiation.cc:769] Sending connection context
0925 02:53:11.651782 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 02:53:11.652083 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260925 02:53:11.654242 19141 heartbeater.cc:347] Connected to a master server at 127.12.222.254:37857
I20260925 02:53:11.654464 19141 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:11.654987 19141 heartbeater.cc:511] Master 127.12.222.254:37857 requested a full tablet report, sending...
I20260925 02:53:11.655985 18811 ts_manager.cc:194] Registered new tserver with Master: 9ff45f8398f24c4c954879f70e0f6598 (127.12.222.194:40847)
I20260925 02:53:11.656776 18811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:42749
WARNING: no policy specified for kudu/127.12.222.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
I20260925 02:53:11.675196 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:37857
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:35211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:11.779409 19146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:11.779630 19146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:11.779670 19146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:11.783493 19146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:11.783540 19146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:11.783591 19146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:53:11.788151 19146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:35211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:11.789196 19146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:11.790099 19146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:11.799063 19152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:11.799220 19146 server_base.cc:1034] running on GCE node
W20260925 02:53:11.799077 19151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:11.799386 19154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:11.799752 19146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:11.800415 19146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:11.801594 19146 hybrid_clock.cc:648] HybridClock initialized: now 1790304791801577 us; error 28 us; skew 500 ppm
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[18766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304791, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:11.804198 19146 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:53:11.804991 19146 webserver.cc:460] Webserver started at http://127.12.222.195:43663/ using document root <none> and password file <none>
I20260925 02:53:11.805472 19146 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:11.805527 19146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:11.805670 19146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:11.807443 19146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "ef768de80e1549c0bf78f3d834ded7ce"
format_stamp: "Formatted at 2026-09-25 02:53:11 on dist-test-slave-252r"
server_key: "a6f29f1f67f96e12b0f9c63d299e3789"
server_key_iv: "64f79585e49daba86661b91fd0716ea1"
server_key_version: "encryptionkey@0"
I20260925 02:53:11.807917 19146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "ef768de80e1549c0bf78f3d834ded7ce"
format_stamp: "Formatted at 2026-09-25 02:53:11 on dist-test-slave-252r"
server_key: "a6f29f1f67f96e12b0f9c63d299e3789"
server_key_iv: "64f79585e49daba86661b91fd0716ea1"
server_key_version: "encryptionkey@0"
I20260925 02:53:11.811195 19146 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:53:11.813501 19161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:11.814601 19146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:53:11.814723 19146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "ef768de80e1549c0bf78f3d834ded7ce"
format_stamp: "Formatted at 2026-09-25 02:53:11 on dist-test-slave-252r"
server_key: "a6f29f1f67f96e12b0f9c63d299e3789"
server_key_iv: "64f79585e49daba86661b91fd0716ea1"
server_key_version: "encryptionkey@0"
I20260925 02:53:11.814818 19146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:11.844350 19146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:11.847139 19146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:11.847323 19146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:11.847844 19146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:11.848584 19146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:11.848627 19146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:11.848690 19146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:11.848728 19146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:11.857055 19146 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:38129
I20260925 02:53:11.857123 19276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:38129 every 8 connection(s)
I20260925 02:53:11.857869 19146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:53:11.860375 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 19146
I20260925 02:53:11.860504 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:53:11.860790 13179 external_mini_cluster.cc:1428] Setting key 8cd8b5354dd344389ad3ec1703b41da3
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[18766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304791, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:11.870343 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:11.859477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:35833 (local address 127.12.222.254:37857)
0925 02:53:11.859622 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:11.859626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:11.860508 (+ 882us) server_negotiation.cc:407] Connection header received
0925 02:53:11.862230 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:11.862233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:11.862282 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:11.862352 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:11.863473 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.863944 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.865111 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.865357 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.865476 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:11.867626 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:11.867629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:11.867631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:11.867652 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:11.868816 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:11.869236 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:11.869238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:11.869239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:11.869307 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:11.869566 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:11.869568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:11.869569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:11.869706 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:11.869773 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:11.870134 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 02:53:11.870236 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:53:11.871019 19279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:11.859862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37857 (local address 127.12.222.195:35833)
0925 02:53:11.860352 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:11.860408 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:11.862050 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:11.862504 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:11.862509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:11.862938 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:11.863339 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:11.863350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.864060 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.864063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:11.864988 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:11.864997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.865504 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.865507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:11.865666 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:11.866160 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:11.866174 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:11.867484 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:11.868928 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:11.868932 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:11.868940 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:11.869156 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:11.869376 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:11.869378 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:11.869379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:11.869496 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:11.869784 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:11.869788 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:11.869961 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 02:53:11.870090 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 02:53:11.870308 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":99,"threads_started":1}
I20260925 02:53:11.872076 19277 heartbeater.cc:347] Connected to a master server at 127.12.222.254:37857
I20260925 02:53:11.872308 19277 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:11.872702 19277 heartbeater.cc:511] Master 127.12.222.254:37857 requested a full tablet report, sending...
I20260925 02:53:11.873733 18811 ts_manager.cc:194] Registered new tserver with Master: ef768de80e1549c0bf78f3d834ded7ce (127.12.222.195:38129)
I20260925 02:53:11.874264 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:53:11.874437 18811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:35833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/test-admin.keytab.
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[18766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:11.888764 13179 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 02:53:11.891341 19284 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:11.890008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48178 (local address 127.12.222.193:46563)
0925 02:53:11.890387 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:11.890396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:11.890418 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:53:11.890464 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:11.890468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:11.890611 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:11.890719 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:11.891143 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48178: BlockingRecv error: recv got EOF from 127.0.0.1:48178 (error 108)
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":140,"threads_started":1}
W20260925 02:53:11.891564 13179 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/krb5cc)
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[18766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:11.892566 13179 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[18766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:11.893478 13179 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[18766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.193@KRBTEST.COM
I20260925 02:53:11.902974 19284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:11.894337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.12.222.193:46563)
0925 02:53:11.894459 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:11.894463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:11.894554 (+ 91us) server_negotiation.cc:407] Connection header received
0925 02:53:11.894708 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:11.894711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:11.894746 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:11.894828 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:11.895348 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.896074 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.896955 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.897350 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.897467 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:11.899743 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:11.899752 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:11.899761 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:11.899782 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:11.901190 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:11.901638 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:11.901643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:11.901644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:11.901716 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:11.901997 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:11.902000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:11.902002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:11.902226 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:11.902417 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:11.902680 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 02:53:11.902819 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 02:53:11.906252 19284 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:11.905428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48196 (local address 127.12.222.193:46563)
0925 02:53:11.905538 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:11.905541 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:11.905681 (+ 140us) server_negotiation.cc:407] Connection header received
0925 02:53:11.905755 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:11.905756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:11.905787 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:11.905846 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:11.906159 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48196: BlockingRecv error: recv got EOF from 127.0.0.1:48196 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 02:53:11.906394 13179 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/krb5cc' of type FILE
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[18766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:11.907379 13179 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[18766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:11.908321 13179 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[18766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.193@KRBTEST.COM
I20260925 02:53:11.916168 19284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:11.909158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48210 (local address 127.12.222.193:46563)
0925 02:53:11.909267 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:11.909273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:11.909396 (+ 123us) server_negotiation.cc:407] Connection header received
0925 02:53:11.909478 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:11.909480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:11.909510 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:11.909571 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:11.909983 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.910435 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.911098 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.911304 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.911423 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:11.913729 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:11.913732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:11.913733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:11.913751 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:11.914886 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:11.915238 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:11.915241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:11.915242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:11.915306 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:11.915539 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:11.915541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:11.915542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:11.915669 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:11.915750 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:11.915914 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 02:53:11.916017 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 02:53:11.918474 19284 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:11.917592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48220 (local address 127.12.222.193:46563)
0925 02:53:11.917708 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:11.917710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:11.917735 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:53:11.917825 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:11.917827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:11.917855 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:11.917911 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:11.918375 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48220: BlockingRecv error: recv got EOF from 127.0.0.1:48220 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 02:53:11.918526 13179 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304762911399-13179-0/krb5kdc/krb5cc' of type FILE
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[18766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:11.919401 13179 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[18766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:11.920293 13179 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[18766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.193@KRBTEST.COM
I20260925 02:53:11.927268 19284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:11.920992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48224 (local address 127.12.222.193:46563)
0925 02:53:11.921127 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:11.921130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:11.921143 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:53:11.921238 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:11.921240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:11.921268 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:11.921323 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:11.921708 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.922130 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.922756 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:11.922926 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:11.922991 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:11.924884 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:11.924887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:11.924889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:11.924908 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:11.926023 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:11.926368 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:11.926370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:11.926371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:11.926433 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:11.926659 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:11.926661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:11.926662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:11.926790 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:11.926867 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:11.927035 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 02:53:11.927136 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:53:11.928069 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 18873
I20260925 02:53:11.934433 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 19008
I20260925 02:53:11.939339 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 19146
I20260925 02:53:11.944020 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 18779
2026-09-25T02:53:11Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3211 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[19297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[19297](info): set up 2 sockets
Sep 25 02:53:11 dist-test-slave-252r krb5kdc[19297](info): commencing operation
krb5kdc: starting...
W20260925 02:53:12.932658 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:53:13 dist-test-slave-252r krb5kdc[19297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:53:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:13Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.254.keytab.
Entry for principal oryx/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.254.keytab.
I20260925 02:53:13.043218 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:37533
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.254.keytab
--principal=oryx/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:37533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:13.141175 19310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:13.141422 19310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:13.141494 19310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:13.145138 19310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:13.145203 19310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:13.145237 19310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:13.145289 19310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:13.145331 19310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:13.150013 19310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:37533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.254.keytab
--principal=oryx/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:37533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:13.150885 19310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:13.151747 19310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:13.159122 19319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:13.159194 19310 server_base.cc:1034] running on GCE node
W20260925 02:53:13.159109 19316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:13.159155 19317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:13.159893 19310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:13.160786 19310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:13.161966 19310 hybrid_clock.cc:648] HybridClock initialized: now 1790304793161925 us; error 34 us; skew 500 ppm
Sep 25 02:53:13 dist-test-slave-252r krb5kdc[19297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304793, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:13.164186 19310 init.cc:377] Logged in from keytab as oryx/127.12.222.254@KRBTEST.COM (short username oryx)
I20260925 02:53:13.165038 19310 webserver.cc:460] Webserver started at http://127.12.222.254:37881/ using document root <none> and password file <none>
I20260925 02:53:13.165520 19310 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:13.165573 19310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:13.165789 19310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:13.167555 19310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "8b59246fa508453f97fb1c2378658fe2"
format_stamp: "Formatted at 2026-09-25 02:53:13 on dist-test-slave-252r"
server_key: "f34d9b16caea8b03338480284dfd51b7"
server_key_iv: "1a48c472b0b65edbf92b070fae34cdb7"
server_key_version: "encryptionkey@0"
I20260925 02:53:13.168075 19310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "8b59246fa508453f97fb1c2378658fe2"
format_stamp: "Formatted at 2026-09-25 02:53:13 on dist-test-slave-252r"
server_key: "f34d9b16caea8b03338480284dfd51b7"
server_key_iv: "1a48c472b0b65edbf92b070fae34cdb7"
server_key_version: "encryptionkey@0"
I20260925 02:53:13.171304 19310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260925 02:53:13.173372 19326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:13.174387 19310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:53:13.174516 19310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "8b59246fa508453f97fb1c2378658fe2"
format_stamp: "Formatted at 2026-09-25 02:53:13 on dist-test-slave-252r"
server_key: "f34d9b16caea8b03338480284dfd51b7"
server_key_iv: "1a48c472b0b65edbf92b070fae34cdb7"
server_key_version: "encryptionkey@0"
I20260925 02:53:13.174614 19310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:13.192962 19310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:13.195465 19310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:13.195655 19310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:13.202203 19310 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:37533
I20260925 02:53:13.202473 19378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:37533 every 8 connection(s)
I20260925 02:53:13.203022 19310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:13.205425 19379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:13.208201 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 19310
I20260925 02:53:13.208374 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:53:13.208638 13179 external_mini_cluster.cc:1428] Setting key d967b13ce0c0a12919aeaa0267d77b9d
I20260925 02:53:13.210542 19379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2: Bootstrap starting.
I20260925 02:53:13.212841 19379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:13.213621 19379 log.cc:824] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:13.215932 19379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2: No bootstrap required, opened a new log
Sep 25 02:53:13 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM
I20260925 02:53:13.220369 19379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } }
I20260925 02:53:13.220639 19379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:13.220706 19379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b59246fa508453f97fb1c2378658fe2, State: Initialized, Role: FOLLOWER
I20260925 02:53:13.221200 19379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } }
I20260925 02:53:13.221393 19379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:13.221464 19379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:13.221554 19379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:13.222560 19379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } }
I20260925 02:53:13.222651 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:13.209843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52708 (local address 127.12.222.254:37533)
0925 02:53:13.210194 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:13.210208 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:13.210237 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:53:13.210820 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:13.210836 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:13.211121 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:13.211356 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:13.212281 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:13.213212 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:13.214325 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:13.214829 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:13.214964 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:13.217671 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:13.217677 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:13.217692 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:13.217727 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:13.219586 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:13.219996 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:13.220002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:13.220007 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:13.220143 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:13.220462 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:13.220465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:13.220466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:13.220971 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:13.221158 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:13.221405 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 02:53:13.221711 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":116,"threads_started":1}
I20260925 02:53:13.222896 19379 leader_election.cc:302] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b59246fa508453f97fb1c2378658fe2; no voters:
I20260925 02:53:13.223220 19379 leader_election.cc:288] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:53:13.223286 19384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:13.223486 19384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [term 1 LEADER]: Becoming Leader. State: Replica: 8b59246fa508453f97fb1c2378658fe2, State: Running, Role: LEADER
I20260925 02:53:13.223757 19384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } }
I20260925 02:53:13.224449 19379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:13.225526 19386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b59246fa508453f97fb1c2378658fe2. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } }
I20260925 02:53:13.225524 19385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } }
I20260925 02:53:13.225844 19385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:13.225872 19386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:13.226567 19395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:13.229125 19395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:13.233742 19395 catalog_manager.cc:1347] Generated new cluster ID: 606b43d4e17a4d11849e00d1e83f089e
I20260925 02:53:13.233798 19395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:13.254536 19395 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:13.255671 19395 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:13.264093 19395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2: Generated new TSK 0
I20260925 02:53:13.264766 19395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.193.keytab.
Entry for principal oryx/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.193.keytab.
I20260925 02:53:13.289392 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:37533
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.193.keytab
--principal=oryx/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:13.396201 19406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:13.396441 19406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:13.396510 19406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:13.400024 19406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:13.400094 19406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:13.400168 19406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:53:13.404918 19406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.193.keytab
--principal=oryx/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:37533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:13.405788 19406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:13.406690 19406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:13.415139 19412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:13.415095 19411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:13.415095 19414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:13.415215 19406 server_base.cc:1034] running on GCE node
I20260925 02:53:13.415733 19406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:13.416375 19406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:13.417533 19406 hybrid_clock.cc:648] HybridClock initialized: now 1790304793417521 us; error 22 us; skew 500 ppm
Sep 25 02:53:13 dist-test-slave-252r krb5kdc[19297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304793, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:13.419998 19406 init.cc:377] Logged in from keytab as oryx/127.12.222.193@KRBTEST.COM (short username oryx)
I20260925 02:53:13.420753 19406 webserver.cc:460] Webserver started at http://127.12.222.193:38229/ using document root <none> and password file <none>
I20260925 02:53:13.421211 19406 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:13.421322 19406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:13.421538 19406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:13.423297 19406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "250e06c6d0614880b13576fa601a059f"
format_stamp: "Formatted at 2026-09-25 02:53:13 on dist-test-slave-252r"
server_key: "90f2c069243c80eaea000afe7cef0da2"
server_key_iv: "4d41471456be07c7d9a05d947a548498"
server_key_version: "encryptionkey@0"
I20260925 02:53:13.423815 19406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "250e06c6d0614880b13576fa601a059f"
format_stamp: "Formatted at 2026-09-25 02:53:13 on dist-test-slave-252r"
server_key: "90f2c069243c80eaea000afe7cef0da2"
server_key_iv: "4d41471456be07c7d9a05d947a548498"
server_key_version: "encryptionkey@0"
I20260925 02:53:13.427403 19406 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 02:53:13.429497 19421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:13.430480 19406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:53:13.430583 19406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "250e06c6d0614880b13576fa601a059f"
format_stamp: "Formatted at 2026-09-25 02:53:13 on dist-test-slave-252r"
server_key: "90f2c069243c80eaea000afe7cef0da2"
server_key_iv: "4d41471456be07c7d9a05d947a548498"
server_key_version: "encryptionkey@0"
I20260925 02:53:13.430671 19406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:13.439419 19406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:13.442366 19406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:13.442504 19406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:13.442998 19406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:13.443706 19406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:13.443748 19406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:13.443821 19406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:13.443852 19406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:13.451840 19406 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:39419
I20260925 02:53:13.451862 19536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:39419 every 8 connection(s)
I20260925 02:53:13.452620 19406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:53:13.455121 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 19406
I20260925 02:53:13.455243 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:53:13.455595 13179 external_mini_cluster.cc:1428] Setting key bad8ea430e16aac0c02a20d456c52788
Sep 25 02:53:13 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304793, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.12.222.194@KRBTEST.COM; defaulting to no policy
I20260925 02:53:13.465574 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:13.454078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:55469 (local address 127.12.222.254:37533)
0925 02:53:13.454192 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:13.454196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:13.454908 (+ 712us) server_negotiation.cc:407] Connection header received
0925 02:53:13.456545 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:13.456548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:13.456599 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:13.456702 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:13.458000 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:13.458530 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:13.459547 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:13.459801 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:13.459876 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:13.462286 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:13.462288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:13.462290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:13.462309 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:13.463517 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:13.463987 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:13.463994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:13.463995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:13.464061 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:13.464505 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:13.464509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:13.464511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:13.464714 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:13.464829 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:13.465278 (+ 449us) server_negotiation.cc:299] Negotiation successful
0925 02:53:13.465402 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:53:13.466154 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:13.454385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.193:55469)
0925 02:53:13.454761 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:13.454797 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:13.456329 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:13.456868 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:13.456873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:13.457268 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:13.457853 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:13.457873 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:13.458667 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:13.458670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:13.459436 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:13.459451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:13.459952 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:13.459959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:13.460148 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:13.460631 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:13.460653 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:13.462134 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:13.463625 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:13.463630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:13.463641 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:13.463877 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:13.464199 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:13.464202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:13.464203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:13.464408 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:13.464838 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:13.464846 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:13.465053 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 02:53:13.465220 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 02:53:13.465507 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.222.194@KRBTEST.COM" created.
I20260925 02:53:13.467242 19537 heartbeater.cc:347] Connected to a master server at 127.12.222.254:37533
I20260925 02:53:13.467442 19537 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:13.467813 19537 heartbeater.cc:511] Master 127.12.222.254:37533 requested a full tablet report, sending...
I20260925 02:53:13.469158 19343 ts_manager.cc:194] Registered new tserver with Master: 250e06c6d0614880b13576fa601a059f (127.12.222.193:39419)
I20260925 02:53:13.470235 19343 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:55469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.194.keytab.
Entry for principal oryx/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.194.keytab.
I20260925 02:53:13.482810 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:37533
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.194.keytab
--principal=oryx/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:13.585219 19542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:13.585451 19542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:13.585531 19542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:13.589100 19542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:13.589162 19542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:13.589234 19542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:53:13.593787 19542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.194.keytab
--principal=oryx/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:37533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:13.594622 19542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:13.595510 19542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:13.604105 19547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:13.604096 19550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:13.604110 19548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:13.604209 19542 server_base.cc:1034] running on GCE node
I20260925 02:53:13.604760 19542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:13.605319 19542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:13.606468 19542 hybrid_clock.cc:648] HybridClock initialized: now 1790304793606453 us; error 28 us; skew 500 ppm
Sep 25 02:53:13 dist-test-slave-252r krb5kdc[19297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304793, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:13.608914 19542 init.cc:377] Logged in from keytab as oryx/127.12.222.194@KRBTEST.COM (short username oryx)
I20260925 02:53:13.609644 19542 webserver.cc:460] Webserver started at http://127.12.222.194:40987/ using document root <none> and password file <none>
I20260925 02:53:13.610101 19542 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:13.610179 19542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:13.610371 19542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:13.612040 19542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "5db6c31d6b604416b3ff9c95d1fa4931"
format_stamp: "Formatted at 2026-09-25 02:53:13 on dist-test-slave-252r"
server_key: "0cc26d2d07120cb7eebf80e0295d8f18"
server_key_iv: "9776466fe8b96ee4386cf819a9169af8"
server_key_version: "encryptionkey@0"
I20260925 02:53:13.612601 19542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "5db6c31d6b604416b3ff9c95d1fa4931"
format_stamp: "Formatted at 2026-09-25 02:53:13 on dist-test-slave-252r"
server_key: "0cc26d2d07120cb7eebf80e0295d8f18"
server_key_iv: "9776466fe8b96ee4386cf819a9169af8"
server_key_version: "encryptionkey@0"
I20260925 02:53:13.615639 19542 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:53:13.617713 19558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:13.618830 19542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:53:13.618919 19542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "5db6c31d6b604416b3ff9c95d1fa4931"
format_stamp: "Formatted at 2026-09-25 02:53:13 on dist-test-slave-252r"
server_key: "0cc26d2d07120cb7eebf80e0295d8f18"
server_key_iv: "9776466fe8b96ee4386cf819a9169af8"
server_key_version: "encryptionkey@0"
I20260925 02:53:13.618988 19542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:13.638067 19542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:13.640703 19542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:13.640862 19542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:13.641317 19542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:13.641955 19542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:13.641996 19542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:13.642030 19542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:13.642043 19542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:13.649585 19542 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:43343
I20260925 02:53:13.649626 19673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:43343 every 8 connection(s)
I20260925 02:53:13.650310 19542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:53:13.658272 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 19542
I20260925 02:53:13.658402 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
Sep 25 02:53:13 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304793, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM
I20260925 02:53:13.658710 13179 external_mini_cluster.cc:1428] Setting key 26e847072d38269dc495aaca0377a532
I20260925 02:53:13.662631 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:13.651775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:44273 (local address 127.12.222.254:37533)
0925 02:53:13.651903 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:13.651906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:13.652587 (+ 681us) server_negotiation.cc:407] Connection header received
0925 02:53:13.653766 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:13.653770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:13.653829 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:13.653923 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:13.654932 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:13.655445 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:13.656489 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:13.656696 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:13.656798 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:13.659006 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:13.659009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:13.659012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:13.659041 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:13.660730 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:13.661212 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:13.661216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:13.661218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:13.661292 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:13.661675 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:13.661679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:13.661682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:13.661901 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:13.662001 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:13.662385 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 02:53:13.662514 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:53:13.663353 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:13.652027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.194:44273)
0925 02:53:13.652449 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:13.652485 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:13.653570 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:13.654078 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:13.654084 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:13.654391 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:13.654782 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:13.654794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:13.655568 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:13.655572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:13.656363 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:13.656372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:13.656830 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:13.656834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:13.656985 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:13.657398 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:13.657413 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:13.658849 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:13.660873 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:13.660878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:13.660889 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:13.661114 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:13.661414 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:13.661418 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:13.661422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:13.661583 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:13.661998 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:13.662003 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:13.662178 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 02:53:13.662312 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 02:53:13.662517 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":110,"threads_started":1}
I20260925 02:53:13.664476 19674 heartbeater.cc:347] Connected to a master server at 127.12.222.254:37533
I20260925 02:53:13.664695 19674 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:13.665150 19674 heartbeater.cc:511] Master 127.12.222.254:37533 requested a full tablet report, sending...
I20260925 02:53:13.666023 19343 ts_manager.cc:194] Registered new tserver with Master: 5db6c31d6b604416b3ff9c95d1fa4931 (127.12.222.194:43343)
I20260925 02:53:13.666745 19343 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:44273
WARNING: no policy specified for oryx/127.12.222.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.222.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.195.keytab.
Entry for principal oryx/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.195.keytab.
I20260925 02:53:13.687006 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:37533
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.195.keytab
--principal=oryx/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:13.788972 19679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:13.789207 19679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:13.789244 19679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:13.792747 19679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:13.792784 19679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:13.792832 19679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:53:13.797268 19679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.195.keytab
--principal=oryx/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:37533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:13.798067 19679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:13.798854 19679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:13.807271 19684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:13.807264 19687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:13.807283 19685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:13.807565 19679 server_base.cc:1034] running on GCE node
I20260925 02:53:13.807896 19679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:13.808525 19679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:13.809695 19679 hybrid_clock.cc:648] HybridClock initialized: now 1790304793809679 us; error 28 us; skew 500 ppm
Sep 25 02:53:13 dist-test-slave-252r krb5kdc[19297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304793, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:13.812060 19679 init.cc:377] Logged in from keytab as oryx/127.12.222.195@KRBTEST.COM (short username oryx)
I20260925 02:53:13.812847 19679 webserver.cc:460] Webserver started at http://127.12.222.195:42657/ using document root <none> and password file <none>
I20260925 02:53:13.813352 19679 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:13.813441 19679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:13.813642 19679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:13.815404 19679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "f20683157c8f4fb1ae28882925a1a4ca"
format_stamp: "Formatted at 2026-09-25 02:53:13 on dist-test-slave-252r"
server_key: "6fb49e00014c93d8574ad3db459ff248"
server_key_iv: "7c0344dfbeca146106e05b38f2dc4e70"
server_key_version: "encryptionkey@0"
I20260925 02:53:13.815999 19679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "f20683157c8f4fb1ae28882925a1a4ca"
format_stamp: "Formatted at 2026-09-25 02:53:13 on dist-test-slave-252r"
server_key: "6fb49e00014c93d8574ad3db459ff248"
server_key_iv: "7c0344dfbeca146106e05b38f2dc4e70"
server_key_version: "encryptionkey@0"
I20260925 02:53:13.819284 19679 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:53:13.821389 19694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:13.822566 19679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:53:13.822703 19679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "f20683157c8f4fb1ae28882925a1a4ca"
format_stamp: "Formatted at 2026-09-25 02:53:13 on dist-test-slave-252r"
server_key: "6fb49e00014c93d8574ad3db459ff248"
server_key_iv: "7c0344dfbeca146106e05b38f2dc4e70"
server_key_version: "encryptionkey@0"
I20260925 02:53:13.822796 19679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:13.836087 19679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:13.838886 19679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:13.839008 19679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:13.839483 19679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:13.840150 19679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:13.840193 19679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:13.840281 19679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:13.840312 19679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:13.848359 19679 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:36325
I20260925 02:53:13.848405 19808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:36325 every 8 connection(s)
I20260925 02:53:13.849104 19679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:53:13.851954 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 19679
I20260925 02:53:13.852041 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:53:13.852355 13179 external_mini_cluster.cc:1428] Setting key 459eb42a2b66b9f27d60f9f16fb5d862
Sep 25 02:53:13 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304793, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM
I20260925 02:53:13.861523 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:13.850588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:47495 (local address 127.12.222.254:37533)
0925 02:53:13.850732 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:13.850736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:13.851408 (+ 672us) server_negotiation.cc:407] Connection header received
0925 02:53:13.852795 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:13.852801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:13.852856 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:13.852945 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:13.854382 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:13.855152 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:13.856282 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:13.856528 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:13.856639 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:13.858620 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:13.858623 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:13.858625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:13.858648 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:13.859924 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:13.860362 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:13.860364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:13.860365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:13.860437 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:13.860705 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:13.860707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:13.860708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:13.860883 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:13.860963 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:13.861334 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 02:53:13.861420 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:53:13.862152 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:13.850882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.195:47495)
0925 02:53:13.851275 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:13.851311 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:13.852556 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:13.853163 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:13.853171 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:13.853597 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:13.854204 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:13.854223 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:13.855278 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:13.855281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:13.856142 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:13.856152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:13.856673 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:13.856689 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:13.856832 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:13.857234 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:13.857248 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:13.858482 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:13.860022 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:13.860025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:13.860034 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:13.860288 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:13.860511 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:13.860513 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:13.860515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:13.860626 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:13.860965 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:13.860969 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:13.861155 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 02:53:13.861278 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 02:53:13.861480 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
I20260925 02:53:13.863050 19809 heartbeater.cc:347] Connected to a master server at 127.12.222.254:37533
I20260925 02:53:13.863255 19809 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:13.863642 19809 heartbeater.cc:511] Master 127.12.222.254:37533 requested a full tablet report, sending...
I20260925 02:53:13.864562 19343 ts_manager.cc:194] Registered new tserver with Master: f20683157c8f4fb1ae28882925a1a4ca (127.12.222.195:36325)
I20260925 02:53:13.865074 19343 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:47495
I20260925 02:53:13.866068 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.222.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.222.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.222.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.253.keytab.
Entry for principal oryx/127.12.222.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.253.keytab.
I20260925 02:53:13.888978 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.253:37839
--webserver_interface=127.12.222.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.253.keytab
--principal=oryx/127.12.222.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:37533,127.12.222.253:37839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:13.987931 19814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:13.988112 19814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:13.988144 19814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:13.991618 19814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:13.991662 19814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:13.991678 19814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:13.991691 19814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:13.991704 19814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:13.996148 19814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:37533,127.12.222.253:37839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.253.keytab
--principal=oryx/127.12.222.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.253:37839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.222.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
W20260925 02:53:13.996299 19814 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.222.254:37533,127.12.222.253:37839'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 02:53:13.996980 19814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:13.997821 19814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:14.005228 19822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:14.005244 19819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:14.005228 19820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:14.005610 19814 server_base.cc:1034] running on GCE node
I20260925 02:53:14.006203 19814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:14.007191 19814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:14.008351 19814 hybrid_clock.cc:648] HybridClock initialized: now 1790304794008329 us; error 38 us; skew 500 ppm
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:14.010691 19814 init.cc:377] Logged in from keytab as oryx/127.12.222.253@KRBTEST.COM (short username oryx)
I20260925 02:53:14.011449 19814 webserver.cc:460] Webserver started at http://127.12.222.253:44609/ using document root <none> and password file <none>
I20260925 02:53:14.011948 19814 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:14.012044 19814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:14.012301 19814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:14.014000 19814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/data/instance:
uuid: "c4d64ba101454cb18a5b487b4b6b676f"
format_stamp: "Formatted at 2026-09-25 02:53:14 on dist-test-slave-252r"
server_key: "2562c14e6f6735712d47306f72aefc9b"
server_key_iv: "7accb7008a5771e8bba744453a44c8ac"
server_key_version: "encryptionkey@0"
I20260925 02:53:14.014473 19814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/wal/instance:
uuid: "c4d64ba101454cb18a5b487b4b6b676f"
format_stamp: "Formatted at 2026-09-25 02:53:14 on dist-test-slave-252r"
server_key: "2562c14e6f6735712d47306f72aefc9b"
server_key_iv: "7accb7008a5771e8bba744453a44c8ac"
server_key_version: "encryptionkey@0"
I20260925 02:53:14.017864 19814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:53:14.020076 19829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:14.021251 19814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:14.021341 19814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/wal
uuid: "c4d64ba101454cb18a5b487b4b6b676f"
format_stamp: "Formatted at 2026-09-25 02:53:14 on dist-test-slave-252r"
server_key: "2562c14e6f6735712d47306f72aefc9b"
server_key_iv: "7accb7008a5771e8bba744453a44c8ac"
server_key_version: "encryptionkey@0"
I20260925 02:53:14.021413 19814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:14.041414 19814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:14.043713 19814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:14.043839 19814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:14.049782 19814 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.253:37839
I20260925 02:53:14.050029 19882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.253:37839 every 8 connection(s)
I20260925 02:53:14.050591 19814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/data/info.pb
I20260925 02:53:14.052888 19883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:14.053627 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 19814
I20260925 02:53:14.056484 19883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.222.253@KRBTEST.COM
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.253@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM
I20260925 02:53:14.066668 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.057980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.12.222.254:37533)
0925 02:53:14.058106 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.058111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.058354 (+ 243us) server_negotiation.cc:407] Connection header received
0925 02:53:14.058510 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.058517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.058562 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.058643 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.059268 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.059739 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.060707 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.060935 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.061047 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.063398 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.063406 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.063408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.063428 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.064947 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.065374 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.065379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.065380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.065470 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.065782 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.065785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.065787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.065974 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.066070 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.066416 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.066519 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:53:14.066919 19888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.057910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:52718)
0925 02:53:14.058235 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.058267 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.058394 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.058709 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.058714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.059018 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.059122 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.059128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.059865 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.059868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.060592 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.060601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.061083 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.061087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.061187 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.061622 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.061646 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.063268 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.065050 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.065059 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.065063 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.065272 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.065574 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.065577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.065579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.065665 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.066083 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.066090 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.066198 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.066363 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.066519 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":35,"thread_start_us":113,"threads_started":1}
I20260925 02:53:14.067159 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.054879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53926 (local address 127.12.222.253:37839)
0925 02:53:14.055283 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.055295 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.055326 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:53:14.055897 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.055911 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.056183 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.056550 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.057275 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.058528 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.059343 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.059779 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.060011 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.062550 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.062556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.062566 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.062589 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.064406 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.064876 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.064881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.064884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.064985 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.065265 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.065269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.065271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.065595 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.065807 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.066041 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.066264 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":277,"thread_start_us":124,"threads_started":1}
I20260925 02:53:14.068250 19883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.253@KRBTEST.COM for oryx/127.12.222.253@KRBTEST.COM
I20260925 02:53:14.074679 19888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.068633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:37839 (local address 127.0.0.1:53936)
0925 02:53:14.068722 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.068733 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.068795 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.068994 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.068996 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.069127 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.069192 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.069196 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.069804 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.069809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.070238 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.070244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.070665 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.070668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.070748 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.071086 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.071096 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.072071 (+ 975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.073388 (+ 1317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.073393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.073395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.073560 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.073806 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.073808 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.073810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.073878 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.074202 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.074205 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.074276 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.074382 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.074525 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 02:53:14.074697 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.068758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.12.222.253:37839)
0925 02:53:14.068844 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.068848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.068859 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:53:14.068880 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.068882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.068907 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.068985 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.069286 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.069712 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.070355 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.070571 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.070664 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.072182 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.072185 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.072187 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.072206 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.073289 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.073645 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.073650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.073652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.073722 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.073963 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.073966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.073968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.074116 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.074201 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.074443 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.074574 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":31}
I20260925 02:53:14.077023 19883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: Bootstrap starting.
I20260925 02:53:14.078763 19883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:14.079357 19883 log.cc:824] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:14.080966 19883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: No bootstrap required, opened a new log
I20260925 02:53:14.083403 19883 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } }
I20260925 02:53:14.083562 19883 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:14.083601 19883 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d64ba101454cb18a5b487b4b6b676f, State: Initialized, Role: FOLLOWER
I20260925 02:53:14.083885 19883 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } }
I20260925 02:53:14.084216 19890 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } } }
I20260925 02:53:14.084383 19890 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:53:14.084880 19883 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: configured and running, proceeding with master startup.
W20260925 02:53:14.088964 19901 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:53:14.089040 19901 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:53:14.090756 19814 master_runner.cc:428] Detected that this master c4d64ba101454cb18a5b487b4b6b676f is joining an existing cluster
I20260925 02:53:14.090811 19814 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.222.253:37839
I20260925 02:53:14.099812 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.093114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.12.222.253:37839)
0925 02:53:14.093185 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.093190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.093204 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:53:14.093361 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.093365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.093408 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.093493 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.093891 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.094333 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.095156 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.095375 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.095475 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.096525 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.096529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.096532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.096553 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.098219 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.098669 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.098675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.098678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.098760 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.099084 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.099089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.099091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.099281 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.099357 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.099577 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.099691 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":33}
I20260925 02:53:14.100041 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.092787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:37839 (local address 127.0.0.1:53950)
0925 02:53:14.093135 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.093147 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.093233 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.093506 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.093508 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.093712 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.093786 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.093791 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.094483 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.094486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.095040 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.095047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.095486 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.095489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.095583 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.095999 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.096012 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.096404 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.098322 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.098325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.098327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.098535 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.098872 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.098876 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.098881 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.098979 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.099350 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.099353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.099424 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.099526 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.099646 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":75,"threads_started":1}
I20260925 02:53:14.100696 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.092913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:52724)
0925 02:53:14.093181 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.093193 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.093279 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.093642 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.093644 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.093854 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.094019 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.094028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.095036 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.095039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.095899 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.095909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.096488 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.096492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.096583 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.097044 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.097058 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.097462 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.099265 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.099267 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.099268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.099420 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.099685 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.099689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.099690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.099783 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.100206 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.100209 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.100320 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.100431 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.100547 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":36,"thread_start_us":81,"threads_started":1}
I20260925 02:53:14.100713 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.092841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52724 (local address 127.12.222.254:37533)
0925 02:53:14.092967 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.092971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.093250 (+ 279us) server_negotiation.cc:407] Connection header received
0925 02:53:14.093412 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.093415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.093457 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.093527 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.094147 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.094889 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.096015 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.096361 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.096468 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.097639 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.097644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.097648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.097678 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.099177 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.099520 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.099523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.099524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.099600 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.099867 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.099871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.099872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.100123 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.100215 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.100484 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.100586 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304793, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.254@KRBTEST.COM for oryx/127.12.222.253@KRBTEST.COM
I20260925 02:53:14.112022 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.104185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.12.222.253:37839)
0925 02:53:14.104296 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.104299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.104604 (+ 305us) server_negotiation.cc:407] Connection header received
0925 02:53:14.104757 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.104759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.104797 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.104859 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.105537 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.105925 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.106984 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.107150 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.107207 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.109159 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.109162 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.109166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.109189 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.110447 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.110864 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.110867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.110869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.110946 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.111206 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.111208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.111210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.111378 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.111485 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.111769 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.111880 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:53:14.112035 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.104121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:37839 (local address 127.0.0.1:53960)
0925 02:53:14.104469 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.104521 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.104642 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.104912 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.104917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.105308 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.105422 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.105429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.106017 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.106020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.106877 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.106886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.107242 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.107245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.107349 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.107757 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.107770 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.109039 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.110551 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.110561 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.110565 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.110770 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.111033 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.111036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.111038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.111117 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.111488 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.111496 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.111596 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.111722 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.111889 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":64,"threads_started":1}
I20260925 02:53:14.113236 19343 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.222.253:37839: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f"
member_type: NON_VOTER
last_known_addr {
host: "127.12.222.253"
port: 37839
}
attrs {
promote: true
}
}
dest_uuid: "8b59246fa508453f97fb1c2378658fe2"
cas_config_opid_index: -1
I20260925 02:53:14.113921 19343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: NON_VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: true } }
I20260925 02:53:14.115418 19911 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c4d64ba101454cb18a5b487b4b6b676f (127.12.222.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: NON_VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: true } } }
I20260925 02:53:14.116557 19857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:14.117445 19911 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.222.253:37839
I20260925 02:53:14.117717 19911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: NON_VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: true } } }
I20260925 02:53:14.117794 19911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:14.117795 19857 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 1 FOLLOWER]: Refusing update from remote peer 8b59246fa508453f97fb1c2378658fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 02:53:14.118333 19910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: NON_VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:14.121853 19814 master.cc:537] Master@127.12.222.253:37839 shutting down...
I20260925 02:53:14.123955 19857 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c4d64ba101454cb18a5b487b4b6b676f (127.12.222.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: NON_VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: true } } }
I20260925 02:53:14.125972 19916 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: NON_VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: true } } }
I20260925 02:53:14.126083 19916 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: This master's current role is: LEARNER
I20260925 02:53:14.126161 19916 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: NON_VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: true } } }
I20260925 02:53:14.126216 19916 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: This master's current role is: LEARNER
I20260925 02:53:14.126271 19890 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b59246fa508453f97fb1c2378658fe2. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: NON_VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: true } } }
I20260925 02:53:14.126338 19890 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: This master's current role is: LEARNER
I20260925 02:53:14.128391 19914 mvcc.cc:205] Tried to move back new op lower bound from 7333088433206480896 to 7333088433045266432. Current Snapshot: MvccSnapshot[applied={T|T < 7333088433077178368}]
I20260925 02:53:14.130841 19814 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 1 LEARNER]: Raft consensus shutting down.
WARNING: no policy specified for oryx/127.12.222.252@KRBTEST.COM; defaulting to no policy
I20260925 02:53:14.131160 19814 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 1 LEARNER]: Raft consensus is shut down!
I20260925 02:53:14.131274 19814 tablet_replica.cc:330] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.222.252@KRBTEST.COM" created.
I20260925 02:53:14.146610 19814 master.cc:552] Master@127.12.222.253:37839 shutdown complete.
I20260925 02:53:14.146647 19814 master_runner.cc:305] Clearing existing system tablet
I20260925 02:53:14.146844 19814 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:14.146883 19814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:14.147877 19814 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:53:14.148795 19923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:14.148977 19814 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:14.149111 19814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:53:14.149255 19814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/wal
uuid: "c4d64ba101454cb18a5b487b4b6b676f"
format_stamp: "Formatted at 2026-09-25 02:53:14 on dist-test-slave-252r"
server_key: "2562c14e6f6735712d47306f72aefc9b"
server_key_iv: "7accb7008a5771e8bba744453a44c8ac"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.222.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.252.keytab.
Entry for principal oryx/127.12.222.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.252.keytab.
I20260925 02:53:14.150769 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.252:46161
--webserver_interface=127.12.222.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.252.keytab
--principal=oryx/127.12.222.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:37533,127.12.222.253:37839,127.12.222.252:46161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:53:14.150877 19814 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:14.153038 19814 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 02:53:14.153138 19814 log.cc:1196] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 02:53:14.153415 19814 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: Deleting consensus metadata
I20260925 02:53:14.153586 19814 master_runner.cc:315] Copying system tablet from 127.12.222.254:37533
I20260925 02:53:14.154763 19814 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: tablet copy: Beginning tablet copy session from remote peer at address 127.12.222.254:37533
I20260925 02:53:14.162276 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.155350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52730 (local address 127.12.222.254:37533)
0925 02:53:14.155497 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.155503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.155671 (+ 168us) server_negotiation.cc:407] Connection header received
0925 02:53:14.155790 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.155793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.155838 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.155923 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.156378 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.156878 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.157813 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.158023 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.158119 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.159198 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.159202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.159204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.159227 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.160673 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.161123 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.161127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.161128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.161209 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.161486 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.161491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.161493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.161660 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.161767 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.162027 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.162131 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:53:14.162288 19931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.155284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:52730)
0925 02:53:14.155573 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.155586 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.155713 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.155945 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.155948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.156126 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.156219 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.156285 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.156990 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.156994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.157683 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.157692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.158131 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.158134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.158217 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.158718 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.158750 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.159053 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.160788 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.160791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.160793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.161004 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.161304 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.161308 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.161310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.161394 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.161757 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.161760 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.161856 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.161970 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.162097 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":93,"threads_started":1}
I20260925 02:53:14.163139 19363 tablet_copy_service.cc:140] P 8b59246fa508453f97fb1c2378658fe2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c4d64ba101454cb18a5b487b4b6b676f ({username='oryx', principal='oryx/127.12.222.253@KRBTEST.COM'} at 127.0.0.1:52730)
I20260925 02:53:14.163300 19363 tablet_copy_service.cc:161] P 8b59246fa508453f97fb1c2378658fe2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c4d64ba101454cb18a5b487b4b6b676f at {username='oryx', principal='oryx/127.12.222.253@KRBTEST.COM'} at 127.0.0.1:52730: session id = c4d64ba101454cb18a5b487b4b6b676f-00000000000000000000000000000000
I20260925 02:53:14.164971 19363 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 02:53:14.166311 19814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 02:53:14.167747 19814 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: tablet copy: Starting download of 0 data blocks...
I20260925 02:53:14.167968 19814 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: tablet copy: Starting download of 1 WAL segments...
I20260925 02:53:14.169454 19814 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 02:53:14.170681 19363 tablet_copy_service.cc:342] P 8b59246fa508453f97fb1c2378658fe2: Request end of tablet copy session c4d64ba101454cb18a5b487b4b6b676f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.222.253@KRBTEST.COM'} at 127.0.0.1:52730
I20260925 02:53:14.170785 19363 tablet_copy_service.cc:433] P 8b59246fa508453f97fb1c2378658fe2: ending tablet copy session c4d64ba101454cb18a5b487b4b6b676f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c4d64ba101454cb18a5b487b4b6b676f
I20260925 02:53:14.177011 19814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:14.178457 19935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:14.178653 19933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:14.178705 19932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:14.178810 19814 server_base.cc:1034] running on GCE node
I20260925 02:53:14.179122 19814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:14.179409 19814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:14.180540 19814 hybrid_clock.cc:648] HybridClock initialized: now 1790304794180531 us; error 23 us; skew 500 ppm
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:14.181847 19814 init.cc:377] Logged in from keytab as oryx/127.12.222.253@KRBTEST.COM (short username oryx)
I20260925 02:53:14.182279 19814 webserver.cc:460] Webserver started at http://127.12.222.253:39265/ using document root <none> and password file <none>
I20260925 02:53:14.182446 19814 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:14.182497 19814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:14.183378 19814 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:53:14.184182 19942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:14.184433 19814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:53:14.184523 19814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/wal
uuid: "c4d64ba101454cb18a5b487b4b6b676f"
format_stamp: "Formatted at 2026-09-25 02:53:14 on dist-test-slave-252r"
server_key: "2562c14e6f6735712d47306f72aefc9b"
server_key_iv: "7accb7008a5771e8bba744453a44c8ac"
server_key_version: "encryptionkey@0"
I20260925 02:53:14.184613 19814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:14.201160 19814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:14.203473 19814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:14.207727 19814 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.253:37839
I20260925 02:53:14.208207 19814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-1/data/info.pb
I20260925 02:53:14.209774 19994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.253:37839 every 8 connection(s)
I20260925 02:53:14.210999 19995 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:53:14.211863 19995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: Bootstrap starting.
I20260925 02:53:14.218137 19995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:53:14.218443 19995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: Bootstrap complete.
I20260925 02:53:14.218917 19995 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: NON_VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: true } }
I20260925 02:53:14.219029 19995 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4d64ba101454cb18a5b487b4b6b676f, State: Initialized, Role: LEARNER
I20260925 02:53:14.219144 19995 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: NON_VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: true } }
I20260925 02:53:14.219554 19995 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:14.223377 20009 catalog_manager.cc:1259] Loaded cluster ID: 606b43d4e17a4d11849e00d1e83f089e
I20260925 02:53:14.223444 20009 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: loading cluster ID for follower catalog manager: success
I20260925 02:53:14.223553 19998 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: NON_VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: true } } }
I20260925 02:53:14.223721 19998 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: This master's current role is: LEARNER
I20260925 02:53:14.225189 20009 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: acquiring CA information for follower catalog manager: success
I20260925 02:53:14.226454 20009 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 02:53:14.288460 19924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:14.288717 19924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:14.288791 19924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:14.292676 19924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:14.292753 19924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:14.292789 19924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:14.292860 19924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:14.292896 19924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:14.297674 19924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:37533,127.12.222.253:37839,127.12.222.252:46161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.252.keytab
--principal=oryx/127.12.222.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.252:46161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.222.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:14.298566 19924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:14.299476 19924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:14.307196 20014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:14.307214 20017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:14.307462 19924 server_base.cc:1034] running on GCE node
W20260925 02:53:14.307466 20015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:14.308127 19924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:14.309162 19924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:14.310331 19924 hybrid_clock.cc:648] HybridClock initialized: now 1790304794310321 us; error 29 us; skew 500 ppm
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:14.312826 19924 init.cc:377] Logged in from keytab as oryx/127.12.222.252@KRBTEST.COM (short username oryx)
I20260925 02:53:14.313563 19924 webserver.cc:460] Webserver started at http://127.12.222.252:39823/ using document root <none> and password file <none>
I20260925 02:53:14.314040 19924 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:14.314122 19924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:14.314327 19924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:14.316156 19924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/data/instance:
uuid: "2c15d4e695f9415bb6ad89272182f911"
format_stamp: "Formatted at 2026-09-25 02:53:14 on dist-test-slave-252r"
server_key: "d00caab7ed222da2dce9b852bbe6663f"
server_key_iv: "d5e6c9a6a9d84d7e834310c8c913e83b"
server_key_version: "encryptionkey@0"
I20260925 02:53:14.316736 19924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/wal/instance:
uuid: "2c15d4e695f9415bb6ad89272182f911"
format_stamp: "Formatted at 2026-09-25 02:53:14 on dist-test-slave-252r"
server_key: "d00caab7ed222da2dce9b852bbe6663f"
server_key_iv: "d5e6c9a6a9d84d7e834310c8c913e83b"
server_key_version: "encryptionkey@0"
I20260925 02:53:14.320276 19924 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:53:14.322700 20024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:14.323863 19924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:14.323998 19924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/wal
uuid: "2c15d4e695f9415bb6ad89272182f911"
format_stamp: "Formatted at 2026-09-25 02:53:14 on dist-test-slave-252r"
server_key: "d00caab7ed222da2dce9b852bbe6663f"
server_key_iv: "d5e6c9a6a9d84d7e834310c8c913e83b"
server_key_version: "encryptionkey@0"
I20260925 02:53:14.324081 19924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:14.356139 19924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:14.358723 19924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:14.358855 19924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:14.365478 19924 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.252:46161
I20260925 02:53:14.365753 20077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.252:46161 every 8 connection(s)
I20260925 02:53:14.366361 19924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/data/info.pb
I20260925 02:53:14.368765 20078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:14.370528 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 19924
I20260925 02:53:14.372120 20078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.222.252@KRBTEST.COM
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.252@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM
I20260925 02:53:14.383380 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.374008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.12.222.254:37533)
0925 02:53:14.374117 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.374121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.374462 (+ 341us) server_negotiation.cc:407] Connection header received
0925 02:53:14.374624 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.374628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.374679 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.374763 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.375506 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.375998 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.376974 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.377234 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.377334 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.379994 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.379997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.380000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.380024 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.381488 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.381987 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.381990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.381992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.382070 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.382495 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.382500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.382504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.382676 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.382784 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.383162 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.383266 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:53:14.384135 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.371991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50030 (local address 127.12.222.252:46161)
0925 02:53:14.372377 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.372394 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.372433 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:53:14.373252 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.373273 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.373634 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.373954 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.374776 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.375661 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.376475 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.376860 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.377018 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.379405 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.379412 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.379432 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.379463 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.381311 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.381773 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.381779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.381781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.381889 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.382180 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.382183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.382185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.382637 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.382798 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.383014 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.383306 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":267,"spinlock_wait_cycles":1408,"thread_start_us":93,"threads_started":1}
I20260925 02:53:14.384135 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.373884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:52740)
0925 02:53:14.374325 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.374363 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.374522 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.374813 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.374819 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.375139 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.375358 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.375368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.376118 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.376121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.376822 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.376831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.377369 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.377373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.377478 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.377944 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.377957 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.379870 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.381597 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.381603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.381609 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.381868 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.382180 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.382183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.382185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.382381 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.382798 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.382804 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.382951 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.383077 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.383350 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":99,"spinlock_wait_cycles":2816,"thread_start_us":129,"threads_started":1}
I20260925 02:53:14.385288 20078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.252@KRBTEST.COM for oryx/127.12.222.253@KRBTEST.COM
I20260925 02:53:14.392452 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.385722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:37839 (local address 127.0.0.1:53968)
0925 02:53:14.385843 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.385854 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.385919 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.386353 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.386354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.386457 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.386526 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.386530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.387305 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.387307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.387804 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.387810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.388221 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.388223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.388345 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.388796 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.388807 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.389831 (+ 1024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.391257 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.391259 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.391261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.391431 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.391663 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.391664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.391666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.391732 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.392035 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.392037 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.392111 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.392179 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.392350 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 02:53:14.392509 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.385823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53968 (local address 127.12.222.253:37839)
0925 02:53:14.386052 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.386056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.386069 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:53:14.386113 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.386118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.386163 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.386245 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.386626 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.387220 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.387893 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.388125 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.388207 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.389962 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.389965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.389967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.389986 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.391167 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.391518 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.391520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.391521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.391589 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.391803 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.391805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.391806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.391959 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.392031 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.392317 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.392402 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
I20260925 02:53:14.393288 20078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.252@KRBTEST.COM for oryx/127.12.222.252@KRBTEST.COM
I20260925 02:53:14.399461 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.393665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:46161 (local address 127.0.0.1:50034)
0925 02:53:14.393741 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.393755 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.393819 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.393988 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.393989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.394096 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.394153 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.394157 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.394782 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.394784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.395240 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.395246 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.395642 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.395644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.395695 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.396025 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.396035 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.397034 (+ 999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.398496 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.398498 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.398499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.398648 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.398853 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.398855 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.398856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.398916 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.399156 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.399158 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.399214 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.399277 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.399355 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 02:53:14.399539 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.393697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50034 (local address 127.12.222.252:46161)
0925 02:53:14.393795 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.393799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.393812 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:53:14.393886 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.393888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.393918 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.393975 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.394232 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.394696 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.395330 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.395542 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.395612 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.397141 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.397144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.397145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.397164 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.398390 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.398721 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.398724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.398725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.398785 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.398967 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.398968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.398969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.399093 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.399149 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.399302 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.399397 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":40}
I20260925 02:53:14.402361 20078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: Bootstrap starting.
I20260925 02:53:14.404606 20078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:14.405293 20078 log.cc:824] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:14.406837 20078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: No bootstrap required, opened a new log
I20260925 02:53:14.409576 20078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } }
I20260925 02:53:14.409725 20078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:14.409792 20078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c15d4e695f9415bb6ad89272182f911, State: Initialized, Role: FOLLOWER
I20260925 02:53:14.410207 20078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } }
I20260925 02:53:14.410607 20084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } } }
I20260925 02:53:14.410727 20084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:53:14.411191 20078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 02:53:14.415642 20095 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:53:14.415712 20095 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:53:14.418473 19924 master_runner.cc:428] Detected that this master 2c15d4e695f9415bb6ad89272182f911 is joining an existing cluster
I20260925 02:53:14.418516 19924 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.222.252:46161
I20260925 02:53:14.428658 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.420790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:37839 (local address 127.0.0.1:53984)
0925 02:53:14.421089 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.421103 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.421216 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.421646 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.421649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.421871 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.422021 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.422027 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.422677 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.422681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.423500 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.423509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.423984 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.423989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.424090 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.425147 (+ 1057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.425162 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.425543 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.427177 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.427181 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.427182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.427450 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.427698 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.427701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.427703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.427792 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.428121 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.428124 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.428248 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.428352 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.428493 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":19,"thread_start_us":108,"threads_started":1}
I20260925 02:53:14.428791 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.420548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.12.222.253:37839)
0925 02:53:14.420643 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.420646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.421161 (+ 515us) server_negotiation.cc:407] Connection header received
0925 02:53:14.421340 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.421344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.421405 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.421491 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.422142 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.422581 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.423614 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.423851 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.423948 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.425674 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.425676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.425678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.425696 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.427084 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.427544 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.427547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.427548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.427613 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.427873 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.427875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.427876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.428032 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.428102 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.428514 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.428636 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:53:14.435076 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.421173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.12.222.252:46161)
0925 02:53:14.421260 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.421263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.428527 (+ 7264us) server_negotiation.cc:407] Connection header received
0925 02:53:14.428663 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.428666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.428698 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.428738 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.429288 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.429982 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.430727 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.431074 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.431174 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.432087 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.432090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.432093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.432114 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.433438 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.433966 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.433969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.433970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.434038 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.434347 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.434349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.434350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.434515 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.434586 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.434844 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.434940 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":34}
I20260925 02:53:14.435444 20103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.421352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:46161 (local address 127.0.0.1:50040)
0925 02:53:14.428453 (+ 7101us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.428467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.428574 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.428855 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.428859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.429061 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.429161 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.429167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.430125 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.430127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.430623 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.430630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.431180 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.431182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.431291 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.431677 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.431688 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.431952 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.433539 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.433542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.433544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.433875 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.434136 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.434139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.434141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.434252 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.434602 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.434605 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.434687 (+ 82us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.435187 (+ 500us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.435289 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7011,"mutex_wait_us":16,"thread_start_us":91,"threads_started":1}
I20260925 02:53:14.437184 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 19406
I20260925 02:53:14.445489 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:39419
--local_ip_for_outbound_sockets=127.12.222.193
--tserver_master_addrs=127.12.222.254:37533
--webserver_port=38229
--webserver_interface=127.12.222.193
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.193.keytab
--principal=oryx/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.222.254:37533,127.12.222.253:37839
--tserver_master_addrs=127.12.222.254:37533,127.12.222.253:37839,127.12.222.252:46161 with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:53:14.456813 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.420807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.12.222.254:37533)
0925 02:53:14.420933 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.420939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.420953 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:53:14.420990 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.420992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.421034 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.421108 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.421834 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.422592 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.423887 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.424251 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.424354 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.453281 (+ 28927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.453287 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.453293 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.453326 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.455100 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.455572 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.455575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.455577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.455666 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.455965 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.455968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.455970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.456165 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.456283 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.456562 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.456679 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:53:14.456995 20101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.420387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:52744)
0925 02:53:14.420633 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.420647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.420740 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.421222 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.421224 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.421571 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.421730 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.421737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.422713 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.422716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.423765 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.423777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.424350 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.424354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.424487 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.452719 (+ 28232us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.452733 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.453124 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.455221 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.455226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.455228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.455468 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.455761 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.455764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.455765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.455867 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.456277 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.456280 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.456381 (+ 101us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.456493 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.456616 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":56,"thread_start_us":99,"threads_started":1}
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304793, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.254@KRBTEST.COM for oryx/127.12.222.252@KRBTEST.COM
I20260925 02:53:14.468647 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.460434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:46161 (local address 127.0.0.1:50050)
0925 02:53:14.460642 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.460653 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.460717 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.460990 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.460992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.461151 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.461237 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.461244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.462109 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.462112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.462886 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.462895 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.463447 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.463451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.463578 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.463973 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.463997 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.465473 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.467211 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.467214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.467216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.467446 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.467722 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.467725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.467727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.467815 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.468173 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.468176 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.468282 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.468379 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.468507 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260925 02:53:14.468648 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.460644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50050 (local address 127.12.222.252:46161)
0925 02:53:14.460735 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.460738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.460762 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:53:14.460814 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.460828 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.460886 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.460960 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.461346 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.462000 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.463016 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.463349 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.463441 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.465630 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.465633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.465635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.465659 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.467106 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.467543 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.467546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.467548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.467625 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.467903 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.467906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.467907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.468077 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.468161 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.468440 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.468532 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:53:14.469795 19343 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.222.252:46161: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2c15d4e695f9415bb6ad89272182f911"
member_type: NON_VOTER
last_known_addr {
host: "127.12.222.252"
port: 46161
}
attrs {
promote: true
}
}
dest_uuid: "8b59246fa508453f97fb1c2378658fe2"
cas_config_opid_index: 5
I20260925 02:53:14.470333 19343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: NON_VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: true } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: NON_VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: true } }
I20260925 02:53:14.471235 19910 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2c15d4e695f9415bb6ad89272182f911 (127.12.222.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: NON_VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: true } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: NON_VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: true } } }
I20260925 02:53:14.472697 19910 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.222.252:46161
I20260925 02:53:14.472860 19910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: NON_VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: true } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: NON_VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: true } } }
I20260925 02:53:14.472949 19910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:14.473073 19924 master.cc:537] Master@127.12.222.252:46161 shutting down...
I20260925 02:53:14.473448 20051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:14.474777 20051 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 1 FOLLOWER]: Refusing update from remote peer 8b59246fa508453f97fb1c2378658fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 02:53:14.475380 19911 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: NON_VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:53:14.476620 19328 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 02:53:14.477538 19328 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 -> Peer 2c15d4e695f9415bb6ad89272182f911 (127.12.222.252:46161): Couldn't send request to peer 2c15d4e695f9415bb6ad89272182f911. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 02:53:14.479933 19924 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:14.480276 19924 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:14.480454 19924 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: stopping tablet replica
I20260925 02:53:14.483088 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.472557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:37839 (local address 127.0.0.1:53994)
0925 02:53:14.478041 (+ 5484us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.478052 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.478144 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.478956 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.478959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.478977 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:14.479062 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.479067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.480491 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.480494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.481669 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.481680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.482724 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.482727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.482814 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.482852 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.482920 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.482981 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5426}
I20260925 02:53:14.483458 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.475209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53994 (local address 127.12.222.253:37839)
0925 02:53:14.477963 (+ 2754us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.477968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.478216 (+ 248us) server_negotiation.cc:407] Connection header received
0925 02:53:14.478260 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.478263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.478319 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.478399 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:14.479618 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.480360 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.481805 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.482436 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.482543 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.482665 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.483278 (+ 613us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.483344 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2679}
I20260925 02:53:14.483811 19969 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 1 LEARNER]: Refusing update from remote peer 8b59246fa508453f97fb1c2378658fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 02:53:14.484164 19910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: NON_VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 02:53:14.485212 19969 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2c15d4e695f9415bb6ad89272182f911 (127.12.222.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: NON_VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: true } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: NON_VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: true } } }
I20260925 02:53:14.485922 19924 master.cc:552] Master@127.12.222.252:46161 shutdown complete.
I20260925 02:53:14.486035 19924 master_runner.cc:305] Clearing existing system tablet
I20260925 02:53:14.486308 19924 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:14.486390 19924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:14.486666 19998 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b59246fa508453f97fb1c2378658fe2. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: NON_VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: true } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: NON_VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: true } } }
I20260925 02:53:14.486768 19998 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: This master's current role is: LEARNER
I20260925 02:53:14.486835 19998 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: NON_VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: true } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: NON_VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: true } } }
I20260925 02:53:14.486896 19998 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: This master's current role is: LEARNER
I20260925 02:53:14.487739 19910 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2: attempting to promote NON_VOTER c4d64ba101454cb18a5b487b4b6b676f to VOTER
I20260925 02:53:14.487778 19924 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:53:14.488089 19910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: NON_VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: true } }
I20260925 02:53:14.489428 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.489169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:46161 (local address 127.0.0.1:50062)
0925 02:53:14.489285 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.489369 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:46161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 02:53:14.489529 19969 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 1 LEARNER]: Refusing update from remote peer 8b59246fa508453f97fb1c2378658fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 02:53:14.490022 19911 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 02:53:14.490561 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.490218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:46161 (local address 127.0.0.1:50070)
0925 02:53:14.490455 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.490511 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:46161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
W20260925 02:53:14.490736 19328 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 -> Peer 2c15d4e695f9415bb6ad89272182f911 (127.12.222.252:46161): Couldn't send request to peer 2c15d4e695f9415bb6ad89272182f911. Status: Network error: Client connection negotiation failed: client connection to 127.12.222.252:46161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 02:53:14.491474 20110 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c4d64ba101454cb18a5b487b4b6b676f (127.12.222.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: NON_VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: true } } }
I20260925 02:53:14.491854 19969 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c4d64ba101454cb18a5b487b4b6b676f (127.12.222.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: NON_VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: true } } }
I20260925 02:53:14.493050 20110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: NON_VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: true } } }
I20260925 02:53:14.493211 20110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:14.493371 20110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: NON_VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: true } } }
I20260925 02:53:14.493342 19998 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: NON_VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: true } } }
I20260925 02:53:14.493453 19998 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:53:14.493525 20110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:14.496374 20109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:14.496603 19924 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:14.496757 19924 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260925 02:53:14.496806 19924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/wal
uuid: "2c15d4e695f9415bb6ad89272182f911"
format_stamp: "Formatted at 2026-09-25 02:53:14 on dist-test-slave-252r"
server_key: "d00caab7ed222da2dce9b852bbe6663f"
server_key_iv: "d5e6c9a6a9d84d7e834310c8c913e83b"
server_key_version: "encryptionkey@0"
I20260925 02:53:14.498951 19924 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:14.501130 19924 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 02:53:14.501215 19924 log.cc:1196] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 02:53:14.501438 19924 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: Deleting consensus metadata
I20260925 02:53:14.501590 19924 master_runner.cc:315] Copying system tablet from 127.12.222.254:37533
I20260925 02:53:14.502573 19924 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: tablet copy: Beginning tablet copy session from remote peer at address 127.12.222.254:37533
I20260925 02:53:14.510926 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.503035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.12.222.254:37533)
0925 02:53:14.503127 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.503131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.503320 (+ 189us) server_negotiation.cc:407] Connection header received
0925 02:53:14.503500 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.503503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.503544 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.503591 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.504162 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.504960 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.506045 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.506383 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.506450 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.507689 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.507692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.507694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.507716 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.509175 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.509672 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.509677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.509680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.509770 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.510089 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.510092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.510094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.510276 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.510341 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.510672 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.510771 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:53:14.511381 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.502979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:52760)
0925 02:53:14.503244 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.503257 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.503379 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.503713 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.503716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.503899 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.504033 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.504039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.505098 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.505100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.505906 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.505915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.506601 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.506604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.506695 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.507185 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.507199 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.507549 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.509292 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.509295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.509296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.509520 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.509882 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.509884 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.509886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.509984 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.510410 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.510413 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.510497 (+ 84us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.511121 (+ 624us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.511239 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":78,"threads_started":1}
I20260925 02:53:14.511744 19363 tablet_copy_service.cc:140] P 8b59246fa508453f97fb1c2378658fe2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2c15d4e695f9415bb6ad89272182f911 ({username='oryx', principal='oryx/127.12.222.252@KRBTEST.COM'} at 127.0.0.1:52760)
I20260925 02:53:14.511827 19363 tablet_copy_service.cc:161] P 8b59246fa508453f97fb1c2378658fe2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2c15d4e695f9415bb6ad89272182f911 at {username='oryx', principal='oryx/127.12.222.252@KRBTEST.COM'} at 127.0.0.1:52760: session id = 2c15d4e695f9415bb6ad89272182f911-00000000000000000000000000000000
I20260925 02:53:14.512781 19363 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 02:53:14.513761 19924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 02:53:14.515663 19924 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: tablet copy: Starting download of 0 data blocks...
I20260925 02:53:14.515811 19924 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: tablet copy: Starting download of 1 WAL segments...
I20260925 02:53:14.517149 19924 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 02:53:14.518743 19363 tablet_copy_service.cc:342] P 8b59246fa508453f97fb1c2378658fe2: Request end of tablet copy session 2c15d4e695f9415bb6ad89272182f911-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.222.252@KRBTEST.COM'} at 127.0.0.1:52760
I20260925 02:53:14.518818 19363 tablet_copy_service.cc:433] P 8b59246fa508453f97fb1c2378658fe2: ending tablet copy session 2c15d4e695f9415bb6ad89272182f911-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2c15d4e695f9415bb6ad89272182f911
I20260925 02:53:14.526669 19924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:14.528302 20117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:14.528309 20120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:14.528915 19924 server_base.cc:1034] running on GCE node
W20260925 02:53:14.532395 20118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:14.532636 19924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:14.532910 19924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:14.534049 19924 hybrid_clock.cc:648] HybridClock initialized: now 1790304794534028 us; error 38 us; skew 500 ppm
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:14.535604 19924 init.cc:377] Logged in from keytab as oryx/127.12.222.252@KRBTEST.COM (short username oryx)
I20260925 02:53:14.536031 19924 webserver.cc:460] Webserver started at http://127.12.222.252:39975/ using document root <none> and password file <none>
I20260925 02:53:14.536199 19924 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:14.536306 19924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:14.537590 19924 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:53:14.538673 20127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:14.538923 19924 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:53:14.539038 19924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/wal
uuid: "2c15d4e695f9415bb6ad89272182f911"
format_stamp: "Formatted at 2026-09-25 02:53:14 on dist-test-slave-252r"
server_key: "d00caab7ed222da2dce9b852bbe6663f"
server_key_iv: "d5e6c9a6a9d84d7e834310c8c913e83b"
server_key_version: "encryptionkey@0"
I20260925 02:53:14.539168 19924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:14.569923 19924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:14.571954 19924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:14.576649 19924 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.252:46161
I20260925 02:53:14.577329 20179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.252:46161 every 8 connection(s)
I20260925 02:53:14.577390 19924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/master-2/data/info.pb
I20260925 02:53:14.578768 20180 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:53:14.579537 20180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: Bootstrap starting.
W20260925 02:53:14.583351 20104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:14.583539 20104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:14.583600 20104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 02:53:14.587352 20180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:53:14.587612 20180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: Bootstrap complete.
W20260925 02:53:14.587723 20104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:14.587775 20104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:14.587862 20104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:53:14.588120 20180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: NON_VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: true } }
I20260925 02:53:14.588264 20180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c15d4e695f9415bb6ad89272182f911, State: Initialized, Role: LEARNER
I20260925 02:53:14.588394 20180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: NON_VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: true } }
I20260925 02:53:14.588718 20186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: NON_VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: true } } }
I20260925 02:53:14.588822 20186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:53:14.588860 20180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:14.591202 20198 catalog_manager.cc:1259] Loaded cluster ID: 606b43d4e17a4d11849e00d1e83f089e
I20260925 02:53:14.591262 20198 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: loading cluster ID for follower catalog manager: success
I20260925 02:53:14.592650 20198 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: acquiring CA information for follower catalog manager: success
I20260925 02:53:14.593448 20104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.193.keytab
--principal=oryx/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:39419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=38229
--tserver_master_addrs=127.12.222.254:37533,127.12.222.253:37839,127.12.222.252:46161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:14.593650 20198 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:53:14.594318 20104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:14.595230 20104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:14.603729 20203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:14.603775 20201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:14.603729 20200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:14.603791 20104 server_base.cc:1034] running on GCE node
I20260925 02:53:14.604395 20104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:14.604986 20104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:14.606148 20104 hybrid_clock.cc:648] HybridClock initialized: now 1790304794606121 us; error 38 us; skew 500 ppm
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:14.608752 20104 init.cc:377] Logged in from keytab as oryx/127.12.222.193@KRBTEST.COM (short username oryx)
I20260925 02:53:14.609542 20104 webserver.cc:460] Webserver started at http://127.12.222.193:38229/ using document root <none> and password file <none>
I20260925 02:53:14.610033 20104 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:14.610126 20104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:14.613517 20104 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:53:14.615293 20210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:14.616453 20104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:14.616580 20104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "250e06c6d0614880b13576fa601a059f"
format_stamp: "Formatted at 2026-09-25 02:53:13 on dist-test-slave-252r"
server_key: "90f2c069243c80eaea000afe7cef0da2"
server_key_iv: "4d41471456be07c7d9a05d947a548498"
server_key_version: "encryptionkey@0"
I20260925 02:53:14.617059 20104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:14.632627 20104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:14.635491 20104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:14.635629 20104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:14.636078 20104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:14.636881 20104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:14.636941 20104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:14.636978 20104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:14.636994 20104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:14.646341 20104 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:39419
I20260925 02:53:14.646375 20323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:39419 every 8 connection(s)
I20260925 02:53:14.647362 20104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:53:14.651955 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 20104
I20260925 02:53:14.652176 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 19542
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.252@KRBTEST.COM
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.253@KRBTEST.COM
I20260925 02:53:14.660723 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:43343
--local_ip_for_outbound_sockets=127.12.222.194
--tserver_master_addrs=127.12.222.254:37533
--webserver_port=40987
--webserver_interface=127.12.222.194
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.194.keytab
--principal=oryx/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.222.254:37533,127.12.222.253:37839
--tserver_master_addrs=127.12.222.254:37533,127.12.222.253:37839,127.12.222.252:46161 with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:53:14.663938 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.648996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:37793 (local address 127.12.222.253:37839)
0925 02:53:14.649141 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.649147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.649963 (+ 816us) server_negotiation.cc:407] Connection header received
0925 02:53:14.651078 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.651082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.651152 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.651208 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.653081 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.653857 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.656466 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.656772 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.656836 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.660251 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.660257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.660262 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.660291 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.661982 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.662732 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.662737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.662741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.662828 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.663099 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.663104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.663106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.663318 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.663384 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.663693 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.663798 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:53:14.664270 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.650421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:44033 (local address 127.12.222.252:46161)
0925 02:53:14.651331 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.651346 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.651359 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:53:14.651397 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.651399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.651444 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.652069 (+ 625us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.653859 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.654931 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.656304 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.656642 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.657088 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.660223 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.660248 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.660251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.660277 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.662182 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.662747 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.662752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.662754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.662835 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.663145 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.663150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.663151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.663379 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.663730 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.663797 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.663914 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":77,"threads_started":1}
I20260925 02:53:14.664734 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.650306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:46161 (local address 127.12.222.193:44033)
0925 02:53:14.650806 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.650818 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.650890 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.651683 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.651686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.652263 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.653742 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.653749 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.655068 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.655071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.656135 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.656148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.656755 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.656758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.656869 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.657474 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.657487 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.660026 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.662293 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.662296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.662297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.662517 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.662942 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.662948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.662950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.663030 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.663487 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.663492 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.663580 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.663654 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.663851 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":1505,"thread_start_us":67,"threads_started":1}
I20260925 02:53:14.664973 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.649369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:37839 (local address 127.12.222.193:37793)
0925 02:53:14.649829 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.649875 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.650847 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.651516 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.651524 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.652127 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.652894 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.652913 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.653999 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.654002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.656358 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.656366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.657462 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.657470 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.657575 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.658015 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.658032 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.660102 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.662120 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.662125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.662128 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.662647 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.662925 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.662928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.662929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.663000 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.663457 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.663460 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.663522 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.664182 (+ 660us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.664312 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":133,"spinlock_wait_cycles":2816,"thread_start_us":152,"threads_started":1}
I20260925 02:53:14.665709 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.648993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:50441 (local address 127.12.222.254:37533)
0925 02:53:14.649113 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.649118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.650707 (+ 1589us) server_negotiation.cc:407] Connection header received
0925 02:53:14.651223 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.651227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.651281 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.651366 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.653088 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.653971 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.655834 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.656226 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.656372 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.659284 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.659290 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.659295 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.659325 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.661787 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.662387 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.662390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.662392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.662472 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.662803 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.662806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.662807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.663012 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.663090 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.665458 (+ 2368us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.665566 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:53:14.666123 20324 heartbeater.cc:347] Connected to a master server at 127.12.222.252:46161
I20260925 02:53:14.666157 20331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.649377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.193:50441)
0925 02:53:14.650590 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.650602 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.651104 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.651516 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.651526 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.652143 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.652894 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.652913 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.654103 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.654109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.655695 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.655708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.656531 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.656537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.656750 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.657398 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.657420 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.659135 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.661914 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.661921 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.661934 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.662293 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.662558 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.662561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.662563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.662700 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.663134 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.663141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.663409 (+ 268us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.663579 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.666013 (+ 2434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"mutex_wait_us":253,"spinlock_wait_cycles":5760,"thread_start_us":159,"threads_started":1}
I20260925 02:53:14.666184 20326 heartbeater.cc:347] Connected to a master server at 127.12.222.253:37839
I20260925 02:53:14.666523 20326 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:14.666906 20325 heartbeater.cc:347] Connected to a master server at 127.12.222.254:37533
I20260925 02:53:14.667014 20325 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:14.667073 20326 heartbeater.cc:511] Master 127.12.222.253:37839 requested a full tablet report, sending...
I20260925 02:53:14.667238 20325 heartbeater.cc:511] Master 127.12.222.254:37533 requested a full tablet report, sending...
I20260925 02:53:14.666335 20324 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:14.667618 20324 heartbeater.cc:511] Master 127.12.222.252:46161 requested a full tablet report, sending...
I20260925 02:53:14.668568 19343 ts_manager.cc:194] Re-registered known tserver with Master: 250e06c6d0614880b13576fa601a059f (127.12.222.193:39419)
I20260925 02:53:14.668640 19950 ts_manager.cc:194] Registered new tserver with Master: 250e06c6d0614880b13576fa601a059f (127.12.222.193:39419)
I20260925 02:53:14.668639 20144 ts_manager.cc:194] Registered new tserver with Master: 250e06c6d0614880b13576fa601a059f (127.12.222.193:39419)
I20260925 02:53:14.669277 19343 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:50441
W20260925 02:53:14.768064 20333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:14.768332 20333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:14.768414 20333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:14.771967 20333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:14.772038 20333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:14.772123 20333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:53:14.777030 20333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.194.keytab
--principal=oryx/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:43343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=40987
--tserver_master_addrs=127.12.222.254:37533,127.12.222.253:37839,127.12.222.252:46161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:14.777951 20333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:14.778877 20333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:14.787509 20339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:14.787540 20338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:14.787544 20341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:14.787895 20333 server_base.cc:1034] running on GCE node
I20260925 02:53:14.788290 20333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:14.788975 20333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:14.790175 20333 hybrid_clock.cc:648] HybridClock initialized: now 1790304794790142 us; error 33 us; skew 500 ppm
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:14.792809 20333 init.cc:377] Logged in from keytab as oryx/127.12.222.194@KRBTEST.COM (short username oryx)
I20260925 02:53:14.793565 20333 webserver.cc:460] Webserver started at http://127.12.222.194:40987/ using document root <none> and password file <none>
I20260925 02:53:14.794080 20333 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:14.794186 20333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:14.797791 20333 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:53:14.799588 20348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:14.800851 20333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:14.801026 20333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "5db6c31d6b604416b3ff9c95d1fa4931"
format_stamp: "Formatted at 2026-09-25 02:53:13 on dist-test-slave-252r"
server_key: "0cc26d2d07120cb7eebf80e0295d8f18"
server_key_iv: "9776466fe8b96ee4386cf819a9169af8"
server_key_version: "encryptionkey@0"
I20260925 02:53:14.801527 20333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:14.831293 20333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:14.833880 20333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:14.834072 20333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:14.834713 20333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:14.835518 20333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:14.835587 20333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:14.835657 20333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:14.835705 20333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:14.844120 20333 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:43343
I20260925 02:53:14.844144 20462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:43343 every 8 connection(s)
I20260925 02:53:14.845077 20333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:53:14.847162 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 20333
I20260925 02:53:14.847352 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 19679
I20260925 02:53:14.855192 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:36325
--local_ip_for_outbound_sockets=127.12.222.195
--tserver_master_addrs=127.12.222.254:37533
--webserver_port=42657
--webserver_interface=127.12.222.195
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.195.keytab
--principal=oryx/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.222.254:37533,127.12.222.253:37839
--tserver_master_addrs=127.12.222.254:37533,127.12.222.253:37839,127.12.222.252:46161 with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.252@KRBTEST.COM
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.253@KRBTEST.COM
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM
I20260925 02:53:14.862353 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.848678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:52773 (local address 127.12.222.252:46161)
0925 02:53:14.848809 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.848813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.850425 (+ 1612us) server_negotiation.cc:407] Connection header received
0925 02:53:14.850595 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.850600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.850674 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.850728 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.851390 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.852282 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.853943 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.854414 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.854524 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.857953 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.857958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.857961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.857987 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.859749 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.860506 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.860510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.860512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.860604 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.861064 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.861068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.861071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.861295 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.861365 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.862046 (+ 681us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.862194 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:53:14.862969 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.846768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:45277 (local address 127.12.222.253:37839)
0925 02:53:14.846910 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.846913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.850180 (+ 3267us) server_negotiation.cc:407] Connection header received
0925 02:53:14.850245 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.850249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.850301 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.850381 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.851110 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.851903 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.854014 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.854358 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.854421 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.858494 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.858500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.858505 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.858534 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.861173 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.861650 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.861654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.861655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.861764 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.862088 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.862091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.862092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.862295 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.862389 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.862725 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.862834 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:53:14.863790 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.846768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:37823 (local address 127.12.222.254:37533)
0925 02:53:14.847032 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:14.847037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:14.848733 (+ 1696us) server_negotiation.cc:407] Connection header received
0925 02:53:14.849613 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:14.849617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:14.849674 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:14.849745 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:14.851109 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.851903 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.854017 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.854345 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.854427 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.859277 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:14.859283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:14.859286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:14.859314 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:14.861167 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.862425 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.862432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.862435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.862531 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:14.862863 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:14.862868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:14.862871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:14.863099 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:14.863188 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:14.863556 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 02:53:14.863663 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260925 02:53:14.864118 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.847285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.194:37823)
0925 02:53:14.848595 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.848632 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.849419 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.849895 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.849901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.850275 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.850940 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.850958 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.852091 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.852094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.853860 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.853872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.854636 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.854640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.854938 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.855772 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.855783 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.859117 (+ 3334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.861299 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.861304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.861308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.862282 (+ 974us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.862650 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.862654 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.862659 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.862750 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.863255 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.863259 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.863377 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.863477 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.863609 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"mutex_wait_us":359,"spinlock_wait_cycles":16000,"thread_start_us":133,"threads_started":1}
I20260925 02:53:14.864116 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.847285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:37839 (local address 127.12.222.194:45277)
0925 02:53:14.850057 (+ 2772us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.850072 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.850174 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.850504 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.850507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.850703 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.850940 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.850958 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.852091 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.852094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.853860 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.853872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.854636 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.854640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.854938 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.855722 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.855746 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.858082 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.861280 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.861283 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.861285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.861548 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.861875 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.861877 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.861879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.861973 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.862449 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.862452 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.862555 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.863166 (+ 611us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.863292 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2675,"mutex_wait_us":433,"spinlock_wait_cycles":30592,"thread_start_us":113,"threads_started":1}
I20260925 02:53:14.864647 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:14.848248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:46161 (local address 127.12.222.194:52773)
0925 02:53:14.850356 (+ 2108us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:14.850368 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:14.850500 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:14.850879 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:14.850885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:14.851088 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:14.851237 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.851247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.852428 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.852434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.853800 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:14.853814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:14.854711 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:14.854717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:14.854914 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:14.855899 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:14.855918 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:14.857734 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:14.859850 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.859855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.859866 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.860095 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.860736 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:14.860739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:14.860741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:14.860941 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:14.861446 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:14.861453 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:14.861748 (+ 295us) client_negotiation.cc:769] Sending connection context
0925 02:53:14.862725 (+ 977us) client_negotiation.cc:240] Negotiation successful
0925 02:53:14.863208 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2033,"mutex_wait_us":126,"spinlock_wait_cycles":1408,"thread_start_us":107,"threads_started":1}
I20260925 02:53:14.865353 20464 heartbeater.cc:347] Connected to a master server at 127.12.222.254:37533
I20260925 02:53:14.865433 20465 heartbeater.cc:347] Connected to a master server at 127.12.222.253:37839
I20260925 02:53:14.865571 20464 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:14.865571 20465 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:14.865828 20463 heartbeater.cc:347] Connected to a master server at 127.12.222.252:46161
I20260925 02:53:14.865890 20463 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:14.866135 20464 heartbeater.cc:511] Master 127.12.222.254:37533 requested a full tablet report, sending...
I20260925 02:53:14.866184 20465 heartbeater.cc:511] Master 127.12.222.253:37839 requested a full tablet report, sending...
I20260925 02:53:14.866135 20463 heartbeater.cc:511] Master 127.12.222.252:46161 requested a full tablet report, sending...
I20260925 02:53:14.867156 19343 ts_manager.cc:194] Re-registered known tserver with Master: 5db6c31d6b604416b3ff9c95d1fa4931 (127.12.222.194:43343)
I20260925 02:53:14.867198 19950 ts_manager.cc:194] Registered new tserver with Master: 5db6c31d6b604416b3ff9c95d1fa4931 (127.12.222.194:43343)
I20260925 02:53:14.867198 20144 ts_manager.cc:194] Registered new tserver with Master: 5db6c31d6b604416b3ff9c95d1fa4931 (127.12.222.194:43343)
I20260925 02:53:14.867857 19343 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:37823
W20260925 02:53:14.960323 20470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:14.960521 20470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:14.960557 20470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:14.964038 20470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:14.964076 20470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:14.964121 20470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:53:14.968851 20470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.195.keytab
--principal=oryx/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:36325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=42657
--tserver_master_addrs=127.12.222.254:37533,127.12.222.253:37839,127.12.222.252:46161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:14.969672 20470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:14.970506 20470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:14.979130 20478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:14.979130 20476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:14.979187 20475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:14.979234 20470 server_base.cc:1034] running on GCE node
I20260925 02:53:14.979777 20470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:14.980396 20470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:14.981571 20470 hybrid_clock.cc:648] HybridClock initialized: now 1790304794981558 us; error 32 us; skew 500 ppm
Sep 25 02:53:14 dist-test-slave-252r krb5kdc[19297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:14.984076 20470 init.cc:377] Logged in from keytab as oryx/127.12.222.195@KRBTEST.COM (short username oryx)
I20260925 02:53:14.984881 20470 webserver.cc:460] Webserver started at http://127.12.222.195:42657/ using document root <none> and password file <none>
I20260925 02:53:14.985387 20470 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:14.985487 20470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:14.988826 20470 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:53:14.990862 20485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:14.991986 20470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:14.992074 20470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "f20683157c8f4fb1ae28882925a1a4ca"
format_stamp: "Formatted at 2026-09-25 02:53:13 on dist-test-slave-252r"
server_key: "6fb49e00014c93d8574ad3db459ff248"
server_key_iv: "7c0344dfbeca146106e05b38f2dc4e70"
server_key_version: "encryptionkey@0"
I20260925 02:53:14.992596 20470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:15.007558 20470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:15.009938 20470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:15.010064 20470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:15.010512 20470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:15.011210 20470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:15.011267 20470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:15.011301 20470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:15.011314 20470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:15.019912 20470 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:36325
I20260925 02:53:15.019956 20601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:36325 every 8 connection(s)
I20260925 02:53:15.020977 20470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:53:15.021270 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 20470
Sep 25 02:53:15 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM
Sep 25 02:53:15 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.252@KRBTEST.COM
Sep 25 02:53:15 dist-test-slave-252r krb5kdc[19297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304794, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.253@KRBTEST.COM
I20260925 02:53:15.038262 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.023043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:57153 (local address 127.12.222.254:37533)
0925 02:53:15.023168 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.023173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.025350 (+ 2177us) server_negotiation.cc:407] Connection header received
0925 02:53:15.025774 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.025778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.025825 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.025875 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:15.027819 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.028571 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.030090 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.030418 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.030493 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.033553 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:15.033556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:15.033558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:15.033597 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:15.035862 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.036503 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.036507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.036510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.036612 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.037030 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.037036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.037039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.037240 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:15.037333 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.037996 (+ 663us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.038115 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:53:15.038966 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.024520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50086 (local address 127.12.222.252:46161)
0925 02:53:15.025066 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.025071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.025860 (+ 789us) server_negotiation.cc:407] Connection header received
0925 02:53:15.025905 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.025907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.025951 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.026037 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:15.027302 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.028081 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.030234 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.030576 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.030686 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.033642 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:15.033645 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:15.033649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:15.033678 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:15.035418 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.037337 (+ 1919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.037357 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.037358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.037442 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.037742 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.037748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.037751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.037993 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:15.038113 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.038659 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.038778 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":71,"threads_started":1}
I20260925 02:53:15.039975 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.023448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.195:57153)
0925 02:53:15.025239 (+ 1791us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:15.025254 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:15.025539 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:15.026239 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:15.026247 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:15.026755 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:15.027382 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.027401 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.028739 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.028742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.029965 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.029975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.030679 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.030682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.030878 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.031543 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:15.031561 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:15.033378 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:15.036028 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:15.036035 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:15.036051 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:15.036392 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:15.036745 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:15.036749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:15.036753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:15.036923 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:15.037422 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:15.037428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:15.037704 (+ 276us) client_negotiation.cc:769] Sending connection context
0925 02:53:15.038093 (+ 389us) client_negotiation.cc:240] Negotiation successful
0925 02:53:15.038774 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1706,"mutex_wait_us":27,"spinlock_wait_cycles":17664,"thread_start_us":151,"threads_started":1}
I20260925 02:53:15.041038 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.023165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:44419 (local address 127.12.222.252:46161)
0925 02:53:15.023291 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.023295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.024587 (+ 1292us) server_negotiation.cc:407] Connection header received
0925 02:53:15.025775 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.025778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.025825 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.025875 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:15.027778 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.028557 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.031973 (+ 3416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.032323 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.032403 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.035507 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:15.035512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:15.035516 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:15.035553 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:15.037179 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.039533 (+ 2354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.039539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.039543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.039621 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.039911 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.039913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.039915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.040118 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:15.040196 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.040664 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.040772 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 02:53:15.041857 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.024959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:35483 (local address 127.12.222.253:37839)
0925 02:53:15.026010 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.026024 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.027009 (+ 985us) server_negotiation.cc:407] Connection header received
0925 02:53:15.027051 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.027053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.027096 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.027171 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:15.027708 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.029393 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.031381 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.031712 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.031823 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.035914 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:15.035918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:15.035923 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:15.035951 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:15.037989 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.038448 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.038453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.038457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.038546 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.038847 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.038850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.038851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.039050 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:15.039142 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.041480 (+ 2338us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.041608 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":966,"thread_start_us":83,"threads_started":1}
I20260925 02:53:15.040217 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.024526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:37839 (local address 127.12.222.195:35483)
0925 02:53:15.026814 (+ 2288us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:15.026828 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:15.026933 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:15.027278 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:15.027281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:15.027449 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:15.027582 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.027589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.030414 (+ 2825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.030417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.031235 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.031244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.031830 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.031835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.031959 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.032939 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:15.032956 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:15.035781 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:15.038101 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:15.038104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:15.038106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:15.038332 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:15.038653 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:15.038656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:15.038658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:15.038744 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:15.039186 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:15.039191 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:15.039276 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 02:53:15.039434 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 02:53:15.039571 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2232,"thread_start_us":74,"threads_started":1}
I20260925 02:53:15.041864 20603 heartbeater.cc:347] Connected to a master server at 127.12.222.254:37533
I20260925 02:53:15.042325 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.023446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:46161 (local address 127.12.222.195:44419)
0925 02:53:15.024426 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:15.024495 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:15.025539 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:15.026239 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:15.026247 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:15.027582 (+ 1335us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:15.027693 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.027699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.029520 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.029523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.030323 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.030331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.032556 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.032562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.032678 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.033368 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:15.033379 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:15.035354 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:15.039206 (+ 3852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:15.039209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:15.039210 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:15.039432 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:15.039728 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:15.039730 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:15.039732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:15.039818 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:15.040401 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:15.040406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:15.040495 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 02:53:15.041167 (+ 672us) client_negotiation.cc:240] Negotiation successful
0925 02:53:15.041316 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":140,"threads_started":1}
I20260925 02:53:15.042493 20603 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:15.042891 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.024100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54000 (local address 127.12.222.253:37839)
0925 02:53:15.024811 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.024817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.025327 (+ 510us) server_negotiation.cc:407] Connection header received
0925 02:53:15.026167 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.026171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.026220 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.026274 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:15.028519 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.029243 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.030315 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.030623 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.030687 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.035245 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:15.035249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:15.035253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:15.035283 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:15.037281 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.040435 (+ 3154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.040440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.040441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.040520 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.041347 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.041352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.041353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.041542 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:15.041651 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.042558 (+ 907us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.042687 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614}
I20260925 02:53:15.043007 20603 heartbeater.cc:511] Master 127.12.222.254:37533 requested a full tablet report, sending...
I20260925 02:53:15.043185 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.023903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.12.222.254:37533)
0925 02:53:15.025564 (+ 1661us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.025568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.025582 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:53:15.025639 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.025644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.025692 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.025779 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:15.027206 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.028060 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.030928 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.031277 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.031396 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.033373 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:15.033377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:15.033379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:15.033404 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:15.038890 (+ 5486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.040798 (+ 1908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.040802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.040806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.040944 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.041958 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.041963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.041966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.042199 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:15.042314 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.042933 (+ 619us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.043044 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1595,"thread_start_us":59,"threads_started":1}
I20260925 02:53:15.043910 20604 heartbeater.cc:347] Connected to a master server at 127.12.222.253:37839
I20260925 02:53:15.043989 20604 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:15.044170 20604 heartbeater.cc:511] Master 127.12.222.253:37839 requested a full tablet report, sending...
I20260925 02:53:15.044457 19342 ts_manager.cc:194] Re-registered known tserver with Master: f20683157c8f4fb1ae28882925a1a4ca (127.12.222.195:36325)
I20260925 02:53:15.044893 20602 heartbeater.cc:347] Connected to a master server at 127.12.222.252:46161
I20260925 02:53:15.044972 20602 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:15.045100 20602 heartbeater.cc:511] Master 127.12.222.252:46161 requested a full tablet report, sending...
I20260925 02:53:15.045233 19342 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:57153
I20260925 02:53:15.045753 19950 ts_manager.cc:194] Registered new tserver with Master: f20683157c8f4fb1ae28882925a1a4ca (127.12.222.195:36325)
I20260925 02:53:15.045904 20144 ts_manager.cc:194] Registered new tserver with Master: f20683157c8f4fb1ae28882925a1a4ca (127.12.222.195:36325)
I20260925 02:53:15.046598 19342 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:53:15.049476 19342 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:53:15.061206 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.057373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:46161 (local address 127.0.0.1:50094)
0925 02:53:15.057610 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:15.057622 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:15.057722 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:15.057992 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:15.057995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:15.058011 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:15.058094 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.058099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.058983 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.058986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.059951 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.059962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.060904 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.060906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.060993 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.061008 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 02:53:15.061061 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 02:53:15.061111 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":117,"threads_started":1}
I20260925 02:53:15.061312 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.057406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.12.222.252:46161)
0925 02:53:15.057512 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.057516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.057704 (+ 188us) server_negotiation.cc:407] Connection header received
0925 02:53:15.057803 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.057805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.057877 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.057982 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:15.058227 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.058872 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.060080 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.060663 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.060730 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.060829 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.061112 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.061189 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:53:15.062263 20154 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260925 02:53:15.064563 20186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b59246fa508453f97fb1c2378658fe2. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: NON_VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: true } } }
I20260925 02:53:15.064711 20186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:53:15.065747 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.059782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:39419 (local address 127.0.0.1:40156)
0925 02:53:15.061164 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:15.061176 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:15.061255 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:15.061819 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:15.061822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:15.061841 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:15.061955 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.061961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.063133 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.063135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.064148 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.064158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.065374 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.065378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.065456 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.065475 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 02:53:15.065546 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 02:53:15.065607 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1330,"mutex_wait_us":29,"thread_start_us":68,"threads_started":1}
I20260925 02:53:15.065855 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.060491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40156 (local address 127.12.222.193:39419)
0925 02:53:15.060801 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.060812 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.061349 (+ 537us) server_negotiation.cc:407] Connection header received
0925 02:53:15.061462 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.061469 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.061673 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.061827 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:15.062091 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.063015 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.064289 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.065112 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.065219 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.065317 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.065586 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.065693 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":132,"threads_started":1}
I20260925 02:53:15.066497 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.060066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:36325 (local address 127.0.0.1:36948)
0925 02:53:15.061381 (+ 1315us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:15.061392 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:15.061454 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:15.062492 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:15.062495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:15.062511 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:15.062570 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.062576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.063644 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.063646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.064633 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.064644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.066136 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.066139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.066216 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.066235 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 02:53:15.066307 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 02:53:15.066364 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1253,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260925 02:53:15.066744 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.060169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.12.222.195:36325)
0925 02:53:15.062110 (+ 1941us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.062120 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.062142 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:53:15.062200 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.062204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.062378 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.062502 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:15.062682 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.063529 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.064939 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.065702 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.065959 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.066077 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.066364 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.066454 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1837,"thread_start_us":156,"threads_started":1}
I20260925 02:53:15.067119 20628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.059479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:43343 (local address 127.0.0.1:49506)
0925 02:53:15.060873 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:15.060885 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:15.060990 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:15.063134 (+ 2144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:15.063136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:15.063151 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:15.063215 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.063220 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.064490 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.064494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.065567 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.065575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.066774 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.066777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.066853 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.066873 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 02:53:15.066951 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:53:15.067008 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1322,"thread_start_us":94,"threads_started":1}
I20260925 02:53:15.067476 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.059643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49506 (local address 127.12.222.194:43343)
0925 02:53:15.062782 (+ 3139us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.062792 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.062809 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:53:15.062871 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.062875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.063010 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.063342 (+ 332us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:15.063477 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.064368 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.065708 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.066516 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.066766 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.066894 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.067267 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.067350 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3056,"thread_start_us":96,"threads_started":1}
I20260925 02:53:15.068778 20257 tablet_service.cc:1467] Processing CreateTablet for tablet d49592494727428aab4dad1e473972db (DEFAULT_TABLE table=test-table [id=04039c7813d548cca6844ac722af0ed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:15.069057 20535 tablet_service.cc:1467] Processing CreateTablet for tablet d49592494727428aab4dad1e473972db (DEFAULT_TABLE table=test-table [id=04039c7813d548cca6844ac722af0ed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:15.069770 20257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d49592494727428aab4dad1e473972db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:15.069840 20535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d49592494727428aab4dad1e473972db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:15.069712 20396 tablet_service.cc:1467] Processing CreateTablet for tablet d49592494727428aab4dad1e473972db (DEFAULT_TABLE table=test-table [id=04039c7813d548cca6844ac722af0ed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:15.070497 20396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d49592494727428aab4dad1e473972db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:15.075593 20634 tablet_bootstrap.cc:492] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca: Bootstrap starting.
I20260925 02:53:15.077757 20491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: NON_VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: true } } }
I20260925 02:53:15.077909 20491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:15.078168 20634 tablet_bootstrap.cc:654] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:15.078917 20491 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2: attempting to promote NON_VOTER 2c15d4e695f9415bb6ad89272182f911 to VOTER
I20260925 02:53:15.078969 20634 log.cc:824] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:15.079245 20491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } }
I20260925 02:53:15.079614 20635 tablet_bootstrap.cc:492] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931: Bootstrap starting.
I20260925 02:53:15.080294 20154 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 1 LEARNER]: Refusing update from remote peer 8b59246fa508453f97fb1c2378658fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260925 02:53:15.080470 19969 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 1 FOLLOWER]: Refusing update from remote peer 8b59246fa508453f97fb1c2378658fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260925 02:53:15.080956 20637 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260925 02:53:15.081162 20620 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260925 02:53:15.081497 20635 tablet_bootstrap.cc:654] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:15.082300 20635 log.cc:824] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:15.082613 20637 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 2c15d4e695f9415bb6ad89272182f911 (127.12.222.252) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } } }
I20260925 02:53:15.082875 20636 tablet_bootstrap.cc:492] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f: Bootstrap starting.
I20260925 02:53:15.083091 19969 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 2c15d4e695f9415bb6ad89272182f911 (127.12.222.252) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } } }
I20260925 02:53:15.083446 20634 tablet_bootstrap.cc:492] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca: No bootstrap required, opened a new log
I20260925 02:53:15.083635 20634 ts_tablet_manager.cc:1397] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260925 02:53:15.083994 20637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } } }
I20260925 02:53:15.084091 20637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:15.084141 20620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } } }
I20260925 02:53:15.084187 20637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } } }
I20260925 02:53:15.084270 20620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:15.084296 20637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:15.084707 20636 tablet_bootstrap.cc:654] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:15.084561 20154 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 2c15d4e695f9415bb6ad89272182f911 (127.12.222.252) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } } }
I20260925 02:53:15.085453 20636 log.cc:824] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:15.086542 20186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } } }
I20260925 02:53:15.086643 20186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:53:15.087500 20635 tablet_bootstrap.cc:492] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931: No bootstrap required, opened a new log
I20260925 02:53:15.087697 20635 ts_tablet_manager.cc:1397] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260925 02:53:15.089231 20636 tablet_bootstrap.cc:492] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f: No bootstrap required, opened a new log
I20260925 02:53:15.089432 20636 ts_tablet_manager.cc:1397] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260925 02:53:15.089287 20634 raft_consensus.cc:348] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } }
I20260925 02:53:15.089632 20634 raft_consensus.cc:374] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:15.089696 20634 raft_consensus.cc:729] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20683157c8f4fb1ae28882925a1a4ca, State: Initialized, Role: FOLLOWER
I20260925 02:53:15.090144 20634 consensus_queue.cc:260] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } }
I20260925 02:53:15.091207 20634 ts_tablet_manager.cc:1428] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260925 02:53:15.091995 20635 raft_consensus.cc:348] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } }
I20260925 02:53:15.092208 20635 raft_consensus.cc:374] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:15.092278 20635 raft_consensus.cc:729] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db6c31d6b604416b3ff9c95d1fa4931, State: Initialized, Role: FOLLOWER
I20260925 02:53:15.092311 20603 heartbeater.cc:503] Master 127.12.222.254:37533 was elected leader, sending a full tablet report...
I20260925 02:53:15.092697 20635 consensus_queue.cc:260] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } }
I20260925 02:53:15.093531 20635 ts_tablet_manager.cc:1428] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260925 02:53:15.093657 20636 raft_consensus.cc:348] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } }
I20260925 02:53:15.093827 20464 heartbeater.cc:503] Master 127.12.222.254:37533 was elected leader, sending a full tablet report...
I20260925 02:53:15.093866 20636 raft_consensus.cc:374] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:15.093920 20636 raft_consensus.cc:729] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 250e06c6d0614880b13576fa601a059f, State: Initialized, Role: FOLLOWER
I20260925 02:53:15.094339 20636 consensus_queue.cc:260] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } }
I20260925 02:53:15.095134 20636 ts_tablet_manager.cc:1428] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 02:53:15.095593 20644 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b59246fa508453f97fb1c2378658fe2" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } } }
I20260925 02:53:15.095719 20644 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:53:15.096258 20325 heartbeater.cc:503] Master 127.12.222.254:37533 was elected leader, sending a full tablet report...
W20260925 02:53:15.096793 20466 tablet.cc:2369] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:53:15.148643 20645 raft_consensus.cc:482] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 02:53:15.148874 20328 tablet.cc:2369] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:53:15.148830 20645 raft_consensus.cc:504] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } }
I20260925 02:53:15.149801 20645 leader_election.cc:288] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5db6c31d6b604416b3ff9c95d1fa4931 (127.12.222.194:43343), 250e06c6d0614880b13576fa601a059f (127.12.222.193:39419)
I20260925 02:53:15.153465 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.150184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:46085 (local address 127.12.222.194:43343)
0925 02:53:15.150301 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.150308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.150334 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:53:15.150379 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.150381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.150428 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.150509 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:15.150848 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.151404 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.152471 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.152881 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.153014 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.153064 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.153265 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.153344 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:53:15.153453 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.150021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:43343 (local address 127.12.222.195:46085)
0925 02:53:15.150166 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:15.150181 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:15.150267 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:15.150564 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:15.150567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:15.150587 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:15.150730 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.150737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.151518 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.151524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.152356 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.152365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.153023 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.153026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.153131 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.153153 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:53:15.153218 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 02:53:15.153298 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 02:53:15.153502 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.150335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:50271 (local address 127.12.222.193:39419)
0925 02:53:15.150447 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.150450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.150462 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:53:15.150574 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.150577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.150614 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.150701 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:15.150965 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.151444 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.152542 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.152894 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.153045 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.153095 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.153313 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.153391 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:53:15.153465 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.150287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:39419 (local address 127.12.222.195:50271)
0925 02:53:15.150403 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:15.150414 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:15.150473 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:15.150716 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:15.150718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:15.150732 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:15.150853 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.150859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.151580 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.151582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.152404 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.152433 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.153059 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.153063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.153164 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.153180 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:53:15.153277 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 02:53:15.153349 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 02:53:15.153752 20646 raft_consensus.cc:482] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:15.153898 20646 raft_consensus.cc:504] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } }
I20260925 02:53:15.154080 20416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d49592494727428aab4dad1e473972db" candidate_uuid: "f20683157c8f4fb1ae28882925a1a4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" is_pre_election: true
I20260925 02:53:15.154269 20416 raft_consensus.cc:2454] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20683157c8f4fb1ae28882925a1a4ca in term 0.
I20260925 02:53:15.154289 20278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d49592494727428aab4dad1e473972db" candidate_uuid: "f20683157c8f4fb1ae28882925a1a4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "250e06c6d0614880b13576fa601a059f" is_pre_election: true
I20260925 02:53:15.154538 20278 raft_consensus.cc:2454] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20683157c8f4fb1ae28882925a1a4ca in term 0.
I20260925 02:53:15.154788 20488 leader_election.cc:302] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5db6c31d6b604416b3ff9c95d1fa4931, f20683157c8f4fb1ae28882925a1a4ca; no voters:
I20260925 02:53:15.154944 20646 leader_election.cc:288] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 250e06c6d0614880b13576fa601a059f (127.12.222.193:39419), f20683157c8f4fb1ae28882925a1a4ca (127.12.222.195:36325)
I20260925 02:53:15.155093 20645 raft_consensus.cc:2781] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:53:15.155177 20645 raft_consensus.cc:482] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:53:15.155222 20645 raft_consensus.cc:3037] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:15.156575 20645 raft_consensus.cc:504] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } }
I20260925 02:53:15.156879 20645 leader_election.cc:288] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [CANDIDATE]: Term 1 election: Requested vote from peers 5db6c31d6b604416b3ff9c95d1fa4931 (127.12.222.194:43343), 250e06c6d0614880b13576fa601a059f (127.12.222.193:39419)
I20260925 02:53:15.157199 20416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d49592494727428aab4dad1e473972db" candidate_uuid: "f20683157c8f4fb1ae28882925a1a4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db6c31d6b604416b3ff9c95d1fa4931"
I20260925 02:53:15.157220 20278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d49592494727428aab4dad1e473972db" candidate_uuid: "f20683157c8f4fb1ae28882925a1a4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "250e06c6d0614880b13576fa601a059f"
I20260925 02:53:15.157331 20278 raft_consensus.cc:3037] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:15.157332 20416 raft_consensus.cc:3037] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:15.158506 20278 raft_consensus.cc:2454] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20683157c8f4fb1ae28882925a1a4ca in term 1.
I20260925 02:53:15.158653 20416 raft_consensus.cc:2454] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f20683157c8f4fb1ae28882925a1a4ca in term 1.
I20260925 02:53:15.158947 20488 leader_election.cc:302] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5db6c31d6b604416b3ff9c95d1fa4931, f20683157c8f4fb1ae28882925a1a4ca; no voters:
I20260925 02:53:15.159214 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.155374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:58055 (local address 127.12.222.193:39419)
0925 02:53:15.155480 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.155484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.155496 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:53:15.155533 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.155536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.155578 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.155646 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:15.156012 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.156600 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.158050 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.158625 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.158719 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.158763 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.159055 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.159116 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:53:15.159229 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.155200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:39419 (local address 127.12.222.194:58055)
0925 02:53:15.155333 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:15.155348 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:15.155449 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:15.155750 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:15.155752 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:15.155771 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:15.155892 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.155897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.156712 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.156715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.157912 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.157923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.158821 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.158827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.158920 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.158942 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:53:15.159042 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 02:53:15.159109 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":28}
I20260925 02:53:15.159458 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.155444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:36325 (local address 127.12.222.194:33635)
0925 02:53:15.155586 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:15.155597 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:15.155647 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:15.156120 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:15.156123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:15.156137 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:15.156228 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.156236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.157038 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.157043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.158195 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:15.158206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.159112 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.159115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:15.159210 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.159238 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 02:53:15.159307 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 02:53:15.159376 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":27}
I20260925 02:53:15.159240 20645 raft_consensus.cc:2781] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:15.159704 20645 raft_consensus.cc:686] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [term 1 LEADER]: Becoming Leader. State: Replica: f20683157c8f4fb1ae28882925a1a4ca, State: Running, Role: LEADER
I20260925 02:53:15.159823 20278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d49592494727428aab4dad1e473972db" candidate_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "250e06c6d0614880b13576fa601a059f" is_pre_election: true
I20260925 02:53:15.159941 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.155744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:33635 (local address 127.12.222.195:36325)
0925 02:53:15.155864 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.155868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.155881 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:53:15.155915 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.155920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.155955 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.156021 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:15.156352 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.156929 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.158388 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.158921 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.159019 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.159059 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.159794 (+ 735us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.159859 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 02:53:15.159961 20278 raft_consensus.cc:2379] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5db6c31d6b604416b3ff9c95d1fa4931 in current term 1: Already voted for candidate f20683157c8f4fb1ae28882925a1a4ca in this term.
I20260925 02:53:15.160024 20645 consensus_queue.cc:237] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } }
I20260925 02:53:15.160331 20555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d49592494727428aab4dad1e473972db" candidate_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20683157c8f4fb1ae28882925a1a4ca" is_pre_election: true
I20260925 02:53:15.161063 20351 leader_election.cc:302] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5db6c31d6b604416b3ff9c95d1fa4931; no voters: 250e06c6d0614880b13576fa601a059f, f20683157c8f4fb1ae28882925a1a4ca
I20260925 02:53:15.161303 20646 raft_consensus.cc:2726] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 02:53:15.163260 19343 catalog_manager.cc:5495] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca reported cstate change: term changed from 0 to 1, leader changed from <none> to f20683157c8f4fb1ae28882925a1a4ca (127.12.222.195). New cstate: current_term: 1 leader_uuid: "f20683157c8f4fb1ae28882925a1a4ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } health_report { overall_health: HEALTHY } } }
I20260925 02:53:15.194399 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.191045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.12.222.195:36325)
0925 02:53:15.191147 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.191151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.191186 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:53:15.191358 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.191367 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.191408 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.191505 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:53:15.191748 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.192191 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.193080 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.193286 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.193397 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.193704 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:53:15.194050 (+ 346us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:53:15.194166 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.194238 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.194277 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:53:15.199349 20278 raft_consensus.cc:1261] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f [term 1 FOLLOWER]: Refusing update from remote peer f20683157c8f4fb1ae28882925a1a4ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:15.199385 20416 raft_consensus.cc:1261] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931 [term 1 FOLLOWER]: Refusing update from remote peer f20683157c8f4fb1ae28882925a1a4ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:15.199908 20645 consensus_queue.cc:1035] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:15.200104 20648 consensus_queue.cc:1035] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:15.207187 20651 mvcc.cc:205] Tried to move back new op lower bound from 7333088441131003904 to 7333088440977993728. Current Snapshot: MvccSnapshot[applied={T|T < 7333088441131003904}]
I20260925 02:53:15.211144 20653 mvcc.cc:205] Tried to move back new op lower bound from 7333088441131003904 to 7333088440977993728. Current Snapshot: MvccSnapshot[applied={T|T < 7333088441131003904}]
I20260925 02:53:15.214123 20652 mvcc.cc:205] Tried to move back new op lower bound from 7333088441131003904 to 7333088440977993728. Current Snapshot: MvccSnapshot[applied={T|T < 7333088441131003904 or (T in {7333088441131003904})}]
I20260925 02:53:15.224320 19343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:15.224458 19343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:15.228688 19343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b59246fa508453f97fb1c2378658fe2: Sending DeleteTablet for 3 replicas of tablet d49592494727428aab4dad1e473972db
I20260925 02:53:15.229516 20396 tablet_service.cc:1514] Processing DeleteTablet for tablet d49592494727428aab4dad1e473972db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:15 UTC) from {username='oryx', principal='oryx/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:49506
I20260925 02:53:15.230213 20257 tablet_service.cc:1514] Processing DeleteTablet for tablet d49592494727428aab4dad1e473972db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:15 UTC) from {username='oryx', principal='oryx/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:40156
I20260925 02:53:15.230615 20662 tablet_replica.cc:330] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f: stopping tablet replica
I20260925 02:53:15.230643 20535 tablet_service.cc:1514] Processing DeleteTablet for tablet d49592494727428aab4dad1e473972db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:15 UTC) from {username='oryx', principal='oryx/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:36948
I20260925 02:53:15.231019 20662 raft_consensus.cc:2229] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:15.231058 20663 tablet_replica.cc:330] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca: stopping tablet replica
I20260925 02:53:15.231288 20663 raft_consensus.cc:2229] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:53:15.231407 20662 raft_consensus.cc:2258] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:15.231626 20663 raft_consensus.cc:2258] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:15.232187 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 19310
I20260925 02:53:15.233354 20663 ts_tablet_manager.cc:1905] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:15.234234 20662 ts_tablet_manager.cc:1905] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:15.236342 20663 ts_tablet_manager.cc:1918] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:15.236436 20663 log.cc:1196] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/d49592494727428aab4dad1e473972db
I20260925 02:53:15.236750 20663 ts_tablet_manager.cc:1939] T d49592494727428aab4dad1e473972db P f20683157c8f4fb1ae28882925a1a4ca: Deleting consensus metadata
I20260925 02:53:15.237365 20662 ts_tablet_manager.cc:1918] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:15.237515 20662 log.cc:1196] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-0/wal/wals/d49592494727428aab4dad1e473972db
I20260925 02:53:15.237921 20662 ts_tablet_manager.cc:1939] T d49592494727428aab4dad1e473972db P 250e06c6d0614880b13576fa601a059f: Deleting consensus metadata
W20260925 02:53:15.240654 20486 connection.cc:535] server connection from 127.0.0.1:36948 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:53:15.240855 20660 tablet_replica.cc:330] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931: stopping tablet replica
W20260925 02:53:15.241007 20213 connection.cc:535] server connection from 127.0.0.1:40156 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:53:15.241292 20660 raft_consensus.cc:2229] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:15.241631 20660 raft_consensus.cc:2258] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:15.243062 20660 ts_tablet_manager.cc:1905] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:15.245872 20660 ts_tablet_manager.cc:1918] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:15.246001 20660 log.cc:1196] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/d49592494727428aab4dad1e473972db
I20260925 02:53:15.246304 20660 ts_tablet_manager.cc:1939] T d49592494727428aab4dad1e473972db P 5db6c31d6b604416b3ff9c95d1fa4931: Deleting consensus metadata
W20260925 02:53:15.247438 20349 connection.cc:441] server connection from 127.0.0.1:49506 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49506 (request call id 1) could send its response
I20260925 02:53:15.249714 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.231107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54016 (local address 127.12.222.253:37839)
0925 02:53:15.231211 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.231215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.240513 (+ 9298us) server_negotiation.cc:407] Connection header received
0925 02:53:15.240653 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.240657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.240704 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.240771 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:15.241758 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.242512 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.243674 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.243980 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.244047 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.246463 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:15.246467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:15.246470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:15.246494 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:15.247970 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.248410 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.248414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.248415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.248494 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.248756 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.248759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.248760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.248928 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:15.249054 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.249407 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.249529 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:53:15.257875 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.249062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50114 (local address 127.12.222.252:46161)
0925 02:53:15.249159 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.249163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.249637 (+ 474us) server_negotiation.cc:407] Connection header received
0925 02:53:15.249787 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.249789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.249830 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.249873 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:15.250726 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.251456 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.252553 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.252908 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.252977 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.254454 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:15.254457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:15.254459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:15.254482 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:15.256042 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.256654 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.256658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.256659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.256738 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.257021 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.257025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.257026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.257215 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:15.257305 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.257622 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.257728 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 02:53:15.258148 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.248555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54020 (local address 127.12.222.253:37839)
0925 02:53:15.248646 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.248650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.249108 (+ 458us) server_negotiation.cc:407] Connection header received
0925 02:53:15.249237 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.249240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.249284 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.249354 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:15.250350 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.250967 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.252052 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.252453 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.252523 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.254169 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:15.254173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:15.254175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:15.254197 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:15.256141 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.256735 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.256739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.256740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.256827 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.257163 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.257168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.257172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.257384 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:15.257486 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.257858 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.257986 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:53:15.260468 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:15.230701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50104 (local address 127.12.222.252:46161)
0925 02:53:15.230807 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:15.230811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:15.250879 (+ 20068us) server_negotiation.cc:407] Connection header received
0925 02:53:15.251027 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:15.251032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:15.251081 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:15.251126 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:15.251888 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.252678 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.254294 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:15.255201 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:15.255283 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:15.256908 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:15.256912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:15.256915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:15.256948 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:15.258526 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.259411 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.259414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.259416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.259495 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:15.259733 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:15.259736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:15.259737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:15.259909 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:15.260022 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:15.260177 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 02:53:15.260323 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:53:16.212592 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:16.212083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.194:53821)
0925 02:53:16.212364 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:16.212477 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":79,"threads_started":1}
W20260925 02:53:16.213131 20464 heartbeater.cc:644] Failed to heartbeat to 127.12.222.254:37533 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
I20260925 02:53:16.213766 20671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:16.213343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.193:36865)
0925 02:53:16.213608 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:16.213682 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":61,"threads_started":1}
W20260925 02:53:16.214056 20325 heartbeater.cc:644] Failed to heartbeat to 127.12.222.254:37533 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
I20260925 02:53:16.215126 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:16.214713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.195:33619)
0925 02:53:16.214987 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:16.215054 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
W20260925 02:53:16.215392 20603 heartbeater.cc:644] Failed to heartbeat to 127.12.222.254:37533 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
I20260925 02:53:16.728602 20673 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b59246fa508453f97fb1c2378658fe2)
I20260925 02:53:16.728812 20673 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } }
I20260925 02:53:16.729863 20673 leader_election.cc:288] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b59246fa508453f97fb1c2378658fe2 (127.12.222.254:37533), 2c15d4e695f9415bb6ad89272182f911 (127.12.222.252:46161)
I20260925 02:53:16.729871 20674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b59246fa508453f97fb1c2378658fe2)
I20260925 02:53:16.730048 20674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } }
I20260925 02:53:16.730449 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:16.729954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:43050)
0925 02:53:16.730280 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:16.730357 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":49,"threads_started":1}
W20260925 02:53:16.730628 19943 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
I20260925 02:53:16.731035 20674 leader_election.cc:288] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b59246fa508453f97fb1c2378658fe2 (127.12.222.254:37533), c4d64ba101454cb18a5b487b4b6b676f (127.12.222.253:37839)
I20260925 02:53:16.731465 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:16.731071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:43054)
0925 02:53:16.731259 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:16.731337 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
I20260925 02:53:16.731745 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:16.731511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:43060)
0925 02:53:16.731636 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:16.731686 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260925 02:53:16.731702 20128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
W20260925 02:53:16.731846 19943 leader_election.cc:334] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b59246fa508453f97fb1c2378658fe2 (127.12.222.254:37533): Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
I20260925 02:53:16.732599 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:16.732377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:43070)
0925 02:53:16.732482 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:16.732532 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 02:53:16.732692 20128 leader_election.cc:334] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b59246fa508453f97fb1c2378658fe2 (127.12.222.254:37533): Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
I20260925 02:53:16.734752 20676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:16.730041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:46161 (local address 127.0.0.1:38844)
0925 02:53:16.730421 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:16.730435 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:16.730563 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:16.730818 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:16.730821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:16.730838 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:16.731354 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:16.731369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:16.732910 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:16.732916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:16.733667 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:16.733675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:16.734322 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:16.734326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:16.734417 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:16.734450 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 02:53:16.734535 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 02:53:16.734620 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":63,"threads_started":1}
I20260925 02:53:16.734764 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:16.730240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38844 (local address 127.12.222.252:46161)
0925 02:53:16.730448 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:16.730452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:16.730594 (+ 142us) server_negotiation.cc:407] Connection header received
0925 02:53:16.730651 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:16.730653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:16.730726 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:16.730807 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:16.732113 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:16.732802 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:16.733788 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:16.734205 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:16.734327 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:16.734374 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:16.734553 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 02:53:16.734635 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":68,"threads_started":1}
I20260925 02:53:16.734936 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:16.731187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:37839 (local address 127.0.0.1:39268)
0925 02:53:16.731614 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:16.731626 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:16.731720 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:16.732032 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:16.732035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:16.732057 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:16.732165 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:16.732171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:16.733110 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:16.733114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:16.733814 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:16.733821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:16.734497 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:16.734501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:16.734612 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:16.734629 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 02:53:16.734742 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 02:53:16.734806 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":3,"thread_start_us":73,"threads_started":1}
I20260925 02:53:16.734934 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:16.731256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39268 (local address 127.12.222.253:37839)
0925 02:53:16.731567 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:16.731574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:16.731684 (+ 110us) server_negotiation.cc:407] Connection header received
0925 02:53:16.731821 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:16.731823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:16.731886 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:16.731999 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:16.732322 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:16.733002 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:16.733970 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:16.734377 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:16.734503 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:16.734548 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:16.734756 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 02:53:16.734828 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":98,"threads_started":1}
I20260925 02:53:16.735455 19969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c15d4e695f9415bb6ad89272182f911" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c4d64ba101454cb18a5b487b4b6b676f" is_pre_election: true
I20260925 02:53:16.735510 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c4d64ba101454cb18a5b487b4b6b676f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2c15d4e695f9415bb6ad89272182f911" is_pre_election: true
I20260925 02:53:16.735641 19969 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c15d4e695f9415bb6ad89272182f911 in term 1.
I20260925 02:53:16.735693 20154 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d64ba101454cb18a5b487b4b6b676f in term 1.
I20260925 02:53:16.736009 19944 leader_election.cc:302] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c15d4e695f9415bb6ad89272182f911, c4d64ba101454cb18a5b487b4b6b676f; no voters: 8b59246fa508453f97fb1c2378658fe2
I20260925 02:53:16.736021 20129 leader_election.cc:302] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c15d4e695f9415bb6ad89272182f911, c4d64ba101454cb18a5b487b4b6b676f; no voters: 8b59246fa508453f97fb1c2378658fe2
I20260925 02:53:16.736205 20673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 02:53:16.736276 20673 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8b59246fa508453f97fb1c2378658fe2)
I20260925 02:53:16.736315 20673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:53:16.736426 20674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 02:53:16.736478 20674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8b59246fa508453f97fb1c2378658fe2)
I20260925 02:53:16.736527 20674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:53:16.737227 20673 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } }
I20260925 02:53:16.737378 20674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } }
I20260925 02:53:16.737533 20673 leader_election.cc:288] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [CANDIDATE]: Term 2 election: Requested vote from peers 8b59246fa508453f97fb1c2378658fe2 (127.12.222.254:37533), 2c15d4e695f9415bb6ad89272182f911 (127.12.222.252:46161)
I20260925 02:53:16.737857 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c4d64ba101454cb18a5b487b4b6b676f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2c15d4e695f9415bb6ad89272182f911"
I20260925 02:53:16.737900 20674 leader_election.cc:288] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [CANDIDATE]: Term 2 election: Requested vote from peers 8b59246fa508453f97fb1c2378658fe2 (127.12.222.254:37533), c4d64ba101454cb18a5b487b4b6b676f (127.12.222.253:37839)
I20260925 02:53:16.738010 20154 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c4d64ba101454cb18a5b487b4b6b676f in current term 2: Already voted for candidate 2c15d4e695f9415bb6ad89272182f911 in this term.
I20260925 02:53:16.738026 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:16.737735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:43084)
0925 02:53:16.737843 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:16.737891 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 02:53:16.738152 19969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c15d4e695f9415bb6ad89272182f911" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c4d64ba101454cb18a5b487b4b6b676f"
I20260925 02:53:16.738368 19969 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2c15d4e695f9415bb6ad89272182f911 in current term 2: Already voted for candidate c4d64ba101454cb18a5b487b4b6b676f in this term.
I20260925 02:53:16.738664 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:16.738434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:43086)
0925 02:53:16.738557 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:16.738610 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 02:53:16.738778 20128 leader_election.cc:334] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8b59246fa508453f97fb1c2378658fe2 (127.12.222.254:37533): Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
I20260925 02:53:16.738796 20676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:16.737591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:43078)
0925 02:53:16.737847 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:16.738732 (+ 885us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260925 02:53:16.738883 20128 leader_election.cc:302] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c15d4e695f9415bb6ad89272182f911; no voters: 8b59246fa508453f97fb1c2378658fe2, c4d64ba101454cb18a5b487b4b6b676f
I20260925 02:53:16.739024 20674 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 02:53:16.739391 20676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:16.739216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:43092)
0925 02:53:16.739310 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:16.739350 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 02:53:16.739485 19943 leader_election.cc:334] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8b59246fa508453f97fb1c2378658fe2 (127.12.222.254:37533): Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
I20260925 02:53:16.739557 19943 leader_election.cc:302] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4d64ba101454cb18a5b487b4b6b676f; no voters: 2c15d4e695f9415bb6ad89272182f911, 8b59246fa508453f97fb1c2378658fe2
I20260925 02:53:16.739666 20673 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 02:53:17.214242 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:17.213713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.194:51021)
0925 02:53:17.214052 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:17.214118 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":74,"threads_started":1}
I20260925 02:53:17.215018 20684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:17.214564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.193:53141)
0925 02:53:17.214850 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:17.214909 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":105,"threads_started":1}
I20260925 02:53:17.216346 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:17.215954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.195:46959)
0925 02:53:17.216167 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:17.216215 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
I20260925 02:53:18.215261 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.214820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.194:36791)
0925 02:53:18.215101 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.215167 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":68,"threads_started":1}
I20260925 02:53:18.216223 20687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.215753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.193:35679)
0925 02:53:18.216019 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.216095 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":61,"threads_started":1}
I20260925 02:53:18.217257 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.216901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.195:56073)
0925 02:53:18.217141 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.217188 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":67,"threads_started":1}
I20260925 02:53:18.441974 20689 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:18.442137 20689 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } }
I20260925 02:53:18.442519 20689 leader_election.cc:288] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8b59246fa508453f97fb1c2378658fe2 (127.12.222.254:37533), 2c15d4e695f9415bb6ad89272182f911 (127.12.222.252:46161)
I20260925 02:53:18.442919 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c4d64ba101454cb18a5b487b4b6b676f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2c15d4e695f9415bb6ad89272182f911" is_pre_election: true
I20260925 02:53:18.443054 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.442694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:43312)
0925 02:53:18.442890 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.442938 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260925 02:53:18.443063 20154 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d64ba101454cb18a5b487b4b6b676f in term 2.
I20260925 02:53:18.443383 19944 leader_election.cc:302] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c15d4e695f9415bb6ad89272182f911, c4d64ba101454cb18a5b487b4b6b676f; no voters:
I20260925 02:53:18.443575 20689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 02:53:18.443632 20689 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:53:18.443670 20689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:53:18.443917 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.443671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:43326)
0925 02:53:18.443808 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.443860 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260925 02:53:18.443988 19943 leader_election.cc:334] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8b59246fa508453f97fb1c2378658fe2 (127.12.222.254:37533): Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
I20260925 02:53:18.444912 20689 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } }
I20260925 02:53:18.445277 20689 leader_election.cc:288] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [CANDIDATE]: Term 3 election: Requested vote from peers 8b59246fa508453f97fb1c2378658fe2 (127.12.222.254:37533), 2c15d4e695f9415bb6ad89272182f911 (127.12.222.252:46161)
I20260925 02:53:18.445611 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.445364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:43338)
0925 02:53:18.445477 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.445525 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 02:53:18.445637 20154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c4d64ba101454cb18a5b487b4b6b676f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2c15d4e695f9415bb6ad89272182f911"
I20260925 02:53:18.445761 20154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:53:18.446666 20154 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d64ba101454cb18a5b487b4b6b676f in term 3.
I20260925 02:53:18.446923 19944 leader_election.cc:302] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c15d4e695f9415bb6ad89272182f911, c4d64ba101454cb18a5b487b4b6b676f; no voters:
I20260925 02:53:18.447113 20689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 3 FOLLOWER]: Leader election won for term 3
I20260925 02:53:18.447312 20689 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [term 3 LEADER]: Becoming Leader. State: Replica: c4d64ba101454cb18a5b487b4b6b676f, State: Running, Role: LEADER
I20260925 02:53:18.447418 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.447179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:43354)
0925 02:53:18.447293 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.447333 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 02:53:18.447521 19943 leader_election.cc:334] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8b59246fa508453f97fb1c2378658fe2 (127.12.222.254:37533): Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
I20260925 02:53:18.447566 20689 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } }
I20260925 02:53:18.448632 20692 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4d64ba101454cb18a5b487b4b6b676f. Latest consensus state: current_term: 3 leader_uuid: "c4d64ba101454cb18a5b487b4b6b676f" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } } }
I20260925 02:53:18.448715 20692 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:18.448975 20694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:18.450975 20694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:18.451267 20694 catalog_manager.cc:1259] Loaded cluster ID: 606b43d4e17a4d11849e00d1e83f089e
I20260925 02:53:18.451305 20694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:18.451757 20694 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:18.452075 20694 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: Loaded TSK: 0
I20260925 02:53:18.452726 20694 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:53:18.528128 19950 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:53:18.531189 19950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:53:18.533975 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.533721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:43380)
0925 02:53:18.533849 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.533902 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 02:53:18.534024 20154 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [term 3 FOLLOWER]: Refusing update from remote peer c4d64ba101454cb18a5b487b4b6b676f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 02:53:18.534541 20692 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 02:53:18.534684 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.534464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.0.0.1:43382)
0925 02:53:18.534575 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.534624 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 02:53:18.534770 19943 consensus_peers.cc:487] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f -> Peer 8b59246fa508453f97fb1c2378658fe2 (127.12.222.254:37533): Couldn't send request to peer 8b59246fa508453f97fb1c2378658fe2. Status: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 02:53:18.535899 20696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4d64ba101454cb18a5b487b4b6b676f. Latest consensus state: current_term: 3 leader_uuid: "c4d64ba101454cb18a5b487b4b6b676f" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } } }
I20260925 02:53:18.536034 20696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:53:18.537237 20689 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "c4d64ba101454cb18a5b487b4b6b676f" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } } }
I20260925 02:53:18.537374 20689 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:18.538573 20696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "c4d64ba101454cb18a5b487b4b6b676f" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8b59246fa508453f97fb1c2378658fe2" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 37533 } } peers { permanent_uuid: "c4d64ba101454cb18a5b487b4b6b676f" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 37839 } attrs { promote: false } } peers { permanent_uuid: "2c15d4e695f9415bb6ad89272182f911" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 46161 } attrs { promote: false } } }
I20260925 02:53:18.538682 20696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c15d4e695f9415bb6ad89272182f911 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:53:18.548280 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.543656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59970 (local address 127.12.222.193:39419)
0925 02:53:18.543903 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:18.543907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:18.543923 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:53:18.543976 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:18.543978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:18.544024 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:18.544127 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:18.544551 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.545341 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.546883 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.547514 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.547620 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:18.547682 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:18.548054 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 02:53:18.548134 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":92,"threads_started":1}
I20260925 02:53:18.548308 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.543546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:39419 (local address 127.0.0.1:59970)
0925 02:53:18.543782 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.543795 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:18.543875 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:18.544296 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:18.544299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:18.544316 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:18.544412 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:18.544419 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.545476 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.545479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:18.546730 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:18.546740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.547741 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.547744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:18.547893 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:18.547923 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 02:53:18.548024 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 02:53:18.548135 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":97,"thread_start_us":94,"threads_started":1}
I20260925 02:53:18.548348 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.543182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:36325 (local address 127.0.0.1:36798)
0925 02:53:18.543467 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.543481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:18.543568 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:18.544692 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:18.544697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:18.544716 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:18.544797 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:18.544802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.545737 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.545741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:18.546850 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:18.546875 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.547704 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.547708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:18.547869 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:18.547932 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 02:53:18.548021 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 02:53:18.548142 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":9,"thread_start_us":116,"threads_started":1}
I20260925 02:53:18.548692 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.543977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36798 (local address 127.12.222.195:36325)
0925 02:53:18.544468 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:18.544474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:18.544490 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:53:18.544543 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:18.544546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:18.544590 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:18.544669 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:18.544927 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.545637 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.547011 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.547608 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.547716 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:18.547758 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:18.548479 (+ 721us) server_negotiation.cc:299] Negotiation successful
0925 02:53:18.548561 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":89,"threads_started":1}
I20260925 02:53:18.548870 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.542911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:43343 (local address 127.0.0.1:43262)
0925 02:53:18.543032 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.543044 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:18.543134 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:18.544507 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:18.544510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:18.544526 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:18.544611 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:18.544616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.545475 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.545480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:18.546374 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:18.546385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.547321 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.547324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:18.548601 (+ 1277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:18.548622 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 02:53:18.548696 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:53:18.548775 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":99}
I20260925 02:53:18.548962 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.543055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.12.222.194:43343)
0925 02:53:18.544283 (+ 1228us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:18.544290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:18.544306 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:53:18.544355 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:18.544360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:18.544403 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:18.544500 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:18.544743 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.545355 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.546530 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.547198 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.547333 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:18.547386 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:18.548740 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0925 02:53:18.548825 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1113,"thread_start_us":69,"threads_started":1}
I20260925 02:53:18.549098 20257 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8a58dc91684b5fbc4121e838436b37 (DEFAULT_TABLE table=test-table [id=b83378333b454117948d83ed0c78b5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:18.549393 20257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8a58dc91684b5fbc4121e838436b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:18.551358 20707 tablet_bootstrap.cc:492] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f: Bootstrap starting.
I20260925 02:53:18.551656 20535 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8a58dc91684b5fbc4121e838436b37 (DEFAULT_TABLE table=test-table [id=b83378333b454117948d83ed0c78b5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:18.551683 20396 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8a58dc91684b5fbc4121e838436b37 (DEFAULT_TABLE table=test-table [id=b83378333b454117948d83ed0c78b5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:18.551923 20535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8a58dc91684b5fbc4121e838436b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:18.551947 20396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8a58dc91684b5fbc4121e838436b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:18.553879 20708 tablet_bootstrap.cc:492] T 5a8a58dc91684b5fbc4121e838436b37 P 5db6c31d6b604416b3ff9c95d1fa4931: Bootstrap starting.
I20260925 02:53:18.554375 20709 tablet_bootstrap.cc:492] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca: Bootstrap starting.
I20260925 02:53:18.554764 20708 tablet_bootstrap.cc:654] T 5a8a58dc91684b5fbc4121e838436b37 P 5db6c31d6b604416b3ff9c95d1fa4931: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:18.554776 20707 tablet_bootstrap.cc:654] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:18.555342 20709 tablet_bootstrap.cc:654] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:18.555858 20708 tablet_bootstrap.cc:492] T 5a8a58dc91684b5fbc4121e838436b37 P 5db6c31d6b604416b3ff9c95d1fa4931: No bootstrap required, opened a new log
I20260925 02:53:18.555991 20708 ts_tablet_manager.cc:1397] T 5a8a58dc91684b5fbc4121e838436b37 P 5db6c31d6b604416b3ff9c95d1fa4931: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:53:18.555996 20707 tablet_bootstrap.cc:492] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f: No bootstrap required, opened a new log
I20260925 02:53:18.556077 20707 ts_tablet_manager.cc:1397] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 02:53:18.556443 20708 raft_consensus.cc:348] T 5a8a58dc91684b5fbc4121e838436b37 P 5db6c31d6b604416b3ff9c95d1fa4931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } }
I20260925 02:53:18.556555 20708 raft_consensus.cc:374] T 5a8a58dc91684b5fbc4121e838436b37 P 5db6c31d6b604416b3ff9c95d1fa4931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:18.556586 20709 tablet_bootstrap.cc:492] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca: No bootstrap required, opened a new log
I20260925 02:53:18.556599 20708 raft_consensus.cc:729] T 5a8a58dc91684b5fbc4121e838436b37 P 5db6c31d6b604416b3ff9c95d1fa4931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db6c31d6b604416b3ff9c95d1fa4931, State: Initialized, Role: FOLLOWER
I20260925 02:53:18.556543 20707 raft_consensus.cc:348] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } }
I20260925 02:53:18.556643 20707 raft_consensus.cc:374] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:18.556654 20709 ts_tablet_manager.cc:1397] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:18.556680 20707 raft_consensus.cc:729] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 250e06c6d0614880b13576fa601a059f, State: Initialized, Role: FOLLOWER
I20260925 02:53:18.556733 20708 consensus_queue.cc:260] T 5a8a58dc91684b5fbc4121e838436b37 P 5db6c31d6b604416b3ff9c95d1fa4931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } }
I20260925 02:53:18.556828 20707 consensus_queue.cc:260] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } }
I20260925 02:53:18.557080 20708 ts_tablet_manager.cc:1428] T 5a8a58dc91684b5fbc4121e838436b37 P 5db6c31d6b604416b3ff9c95d1fa4931: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:53:18.557081 20709 raft_consensus.cc:348] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } }
I20260925 02:53:18.557180 20709 raft_consensus.cc:374] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:18.557210 20709 raft_consensus.cc:729] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20683157c8f4fb1ae28882925a1a4ca, State: Initialized, Role: FOLLOWER
I20260925 02:53:18.557346 20709 consensus_queue.cc:260] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } }
I20260925 02:53:18.557858 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.557469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.194:46137)
0925 02:53:18.557746 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.557798 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260925 02:53:18.558490 20709 ts_tablet_manager.cc:1428] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260925 02:53:18.558840 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.558627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.195:59173)
0925 02:53:18.558724 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.558776 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 02:53:18.559902 20465 heartbeater.cc:503] Master 127.12.222.253:37839 was elected leader, sending a full tablet report...
I20260925 02:53:18.560140 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.559855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.194:38829)
0925 02:53:18.560018 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.560082 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260925 02:53:18.560299 20707 ts_tablet_manager.cc:1428] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 02:53:18.560348 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.560123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.195:41451)
0925 02:53:18.560217 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.560290 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 02:53:18.560997 20604 heartbeater.cc:503] Master 127.12.222.253:37839 was elected leader, sending a full tablet report...
I20260925 02:53:18.562480 20687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.562243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.193:49549)
0925 02:53:18.562347 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.562416 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 02:53:18.563169 20687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.562955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.193:39725)
0925 02:53:18.563044 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.563095 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 02:53:18.563673 20326 heartbeater.cc:503] Master 127.12.222.253:37839 was elected leader, sending a full tablet report...
I20260925 02:53:18.676364 20715 raft_consensus.cc:482] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:18.676499 20715 raft_consensus.cc:504] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } }
I20260925 02:53:18.677536 20715 leader_election.cc:288] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5db6c31d6b604416b3ff9c95d1fa4931 (127.12.222.194:43343), f20683157c8f4fb1ae28882925a1a4ca (127.12.222.195:36325)
I20260925 02:53:18.681887 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.677845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:45781 (local address 127.12.222.194:43343)
0925 02:53:18.677980 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:18.677986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:18.678005 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:53:18.678065 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:18.678068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:18.678121 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:18.678208 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:18.678561 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.679273 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.680825 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.681386 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.681507 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:18.681556 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:18.681723 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 02:53:18.681790 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:53:18.681887 20687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.677688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:43343 (local address 127.12.222.193:45781)
0925 02:53:18.677862 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.677879 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:18.677970 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:18.678293 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:18.678296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:18.678311 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:18.678423 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:18.678430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.679379 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.679384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:18.680661 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:18.680672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.681507 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.681510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:18.681602 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:18.681627 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:53:18.681707 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 02:53:18.681771 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":61}
I20260925 02:53:18.682245 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.678083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:34947 (local address 127.12.222.195:36325)
0925 02:53:18.678226 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:18.678230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:18.678413 (+ 183us) server_negotiation.cc:407] Connection header received
0925 02:53:18.678502 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:18.678504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:18.678558 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:18.678612 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:18.678875 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.679543 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.681004 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.681551 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.681639 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:18.681686 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:18.682049 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 02:53:18.682138 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:53:18.682576 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.677972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:36325 (local address 127.12.222.193:34947)
0925 02:53:18.678318 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.678332 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:18.678418 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:18.678647 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:18.678649 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:18.678666 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:18.678769 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:18.678775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.679672 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.679675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:18.680813 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:18.680837 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.681795 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.681798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:18.681896 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:18.681921 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:53:18.682389 (+ 468us) client_negotiation.cc:240] Negotiation successful
0925 02:53:18.682458 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":2,"thread_start_us":154,"threads_started":1}
I20260925 02:53:18.682667 20416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8a58dc91684b5fbc4121e838436b37" candidate_uuid: "250e06c6d0614880b13576fa601a059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" is_pre_election: true
I20260925 02:53:18.682792 20416 raft_consensus.cc:2454] T 5a8a58dc91684b5fbc4121e838436b37 P 5db6c31d6b604416b3ff9c95d1fa4931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 250e06c6d0614880b13576fa601a059f in term 0.
I20260925 02:53:18.682971 20555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8a58dc91684b5fbc4121e838436b37" candidate_uuid: "250e06c6d0614880b13576fa601a059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20683157c8f4fb1ae28882925a1a4ca" is_pre_election: true
I20260925 02:53:18.683145 20555 raft_consensus.cc:2454] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 250e06c6d0614880b13576fa601a059f in term 0.
I20260925 02:53:18.683146 20213 leader_election.cc:302] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 250e06c6d0614880b13576fa601a059f, 5db6c31d6b604416b3ff9c95d1fa4931; no voters:
I20260925 02:53:18.683465 20715 raft_consensus.cc:2781] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:53:18.683538 20715 raft_consensus.cc:482] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:53:18.683574 20715 raft_consensus.cc:3037] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:18.684676 20715 raft_consensus.cc:504] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } }
I20260925 02:53:18.684989 20715 leader_election.cc:288] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [CANDIDATE]: Term 1 election: Requested vote from peers 5db6c31d6b604416b3ff9c95d1fa4931 (127.12.222.194:43343), f20683157c8f4fb1ae28882925a1a4ca (127.12.222.195:36325)
I20260925 02:53:18.685374 20555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8a58dc91684b5fbc4121e838436b37" candidate_uuid: "250e06c6d0614880b13576fa601a059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20683157c8f4fb1ae28882925a1a4ca"
I20260925 02:53:18.685406 20416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8a58dc91684b5fbc4121e838436b37" candidate_uuid: "250e06c6d0614880b13576fa601a059f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db6c31d6b604416b3ff9c95d1fa4931"
I20260925 02:53:18.685529 20555 raft_consensus.cc:3037] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:18.685554 20416 raft_consensus.cc:3037] T 5a8a58dc91684b5fbc4121e838436b37 P 5db6c31d6b604416b3ff9c95d1fa4931 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:18.686362 20416 raft_consensus.cc:2454] T 5a8a58dc91684b5fbc4121e838436b37 P 5db6c31d6b604416b3ff9c95d1fa4931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 250e06c6d0614880b13576fa601a059f in term 1.
I20260925 02:53:18.686411 20555 raft_consensus.cc:2454] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 250e06c6d0614880b13576fa601a059f in term 1.
I20260925 02:53:18.686640 20213 leader_election.cc:302] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 250e06c6d0614880b13576fa601a059f, 5db6c31d6b604416b3ff9c95d1fa4931; no voters:
I20260925 02:53:18.686853 20715 raft_consensus.cc:2781] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:18.687060 20715 raft_consensus.cc:686] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [term 1 LEADER]: Becoming Leader. State: Replica: 250e06c6d0614880b13576fa601a059f, State: Running, Role: LEADER
I20260925 02:53:18.687307 20715 consensus_queue.cc:237] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } }
I20260925 02:53:18.687666 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.687445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.193:36169)
0925 02:53:18.687551 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.687612 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 02:53:18.689751 19952 catalog_manager.cc:5495] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f reported cstate change: term changed from 0 to 1, leader changed from <none> to 250e06c6d0614880b13576fa601a059f (127.12.222.193). New cstate: current_term: 1 leader_uuid: "250e06c6d0614880b13576fa601a059f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "250e06c6d0614880b13576fa601a059f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39419 } health_report { overall_health: HEALTHY } } }
I20260925 02:53:18.733299 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.730486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59984 (local address 127.12.222.193:39419)
0925 02:53:18.730591 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:18.730594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:18.730605 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:53:18.730701 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:18.730703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:18.730747 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:18.730837 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:53:18.731035 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.731457 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.732181 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:18.732377 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:18.732442 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:18.732649 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:53:18.733041 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:53:18.733119 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:18.733175 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 02:53:18.733215 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:53:18.735699 20416 raft_consensus.cc:1261] T 5a8a58dc91684b5fbc4121e838436b37 P 5db6c31d6b604416b3ff9c95d1fa4931 [term 1 FOLLOWER]: Refusing update from remote peer 250e06c6d0614880b13576fa601a059f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:18.735781 20555 raft_consensus.cc:1261] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca [term 1 FOLLOWER]: Refusing update from remote peer 250e06c6d0614880b13576fa601a059f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:18.736179 20715 consensus_queue.cc:1035] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db6c31d6b604416b3ff9c95d1fa4931" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:18.736397 20718 consensus_queue.cc:1035] T 5a8a58dc91684b5fbc4121e838436b37 P 250e06c6d0614880b13576fa601a059f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20683157c8f4fb1ae28882925a1a4ca" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:18.738343 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.737823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.194:45291)
0925 02:53:18.738215 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.738260 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327}
I20260925 02:53:18.739522 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.739243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.195:36051)
0925 02:53:18.739392 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.739454 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 02:53:18.741146 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.740794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.193:53771)
0925 02:53:18.741042 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.741092 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260925 02:53:18.743098 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.742907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.194:44461)
0925 02:53:18.743011 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.743052 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 02:53:18.744771 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.744534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.193:51123)
0925 02:53:18.744667 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.744712 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 02:53:18.745518 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.745281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:37533 (local address 127.12.222.195:40307)
0925 02:53:18.745409 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.745463 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:37533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 02:53:18.749146 19952 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:18.749262 19952 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:18.752660 19952 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4d64ba101454cb18a5b487b4b6b676f: Sending DeleteTablet for 3 replicas of tablet 5a8a58dc91684b5fbc4121e838436b37
I20260925 02:53:18.753305 20535 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8a58dc91684b5fbc4121e838436b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:18 UTC) from {username='oryx', principal='oryx/127.12.222.253@KRBTEST.COM'} at 127.0.0.1:36798
I20260925 02:53:18.753502 20396 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8a58dc91684b5fbc4121e838436b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:18 UTC) from {username='oryx', principal='oryx/127.12.222.253@KRBTEST.COM'} at 127.0.0.1:43262
I20260925 02:53:18.753780 20257 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8a58dc91684b5fbc4121e838436b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:18 UTC) from {username='oryx', principal='oryx/127.12.222.253@KRBTEST.COM'} at 127.0.0.1:59970
I20260925 02:53:18.754001 20731 tablet_replica.cc:330] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca: stopping tablet replica
I20260925 02:53:18.754187 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 20104
I20260925 02:53:18.755537 20731 raft_consensus.cc:2229] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:18.755681 20731 raft_consensus.cc:2258] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:18.756456 20731 ts_tablet_manager.cc:1905] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:18.759595 20731 ts_tablet_manager.cc:1918] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:18.759676 20731 log.cc:1196] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/5a8a58dc91684b5fbc4121e838436b37
I20260925 02:53:18.759980 20731 ts_tablet_manager.cc:1939] T 5a8a58dc91684b5fbc4121e838436b37 P f20683157c8f4fb1ae28882925a1a4ca: Deleting consensus metadata
I20260925 02:53:18.760728 19945 catalog_manager.cc:4841] TS f20683157c8f4fb1ae28882925a1a4ca (127.12.222.195:36325): tablet 5a8a58dc91684b5fbc4121e838436b37 (table test-table [id=b83378333b454117948d83ed0c78b5bc]) successfully deleted
I20260925 02:53:18.762418 20732 tablet_replica.cc:330] T 5a8a58dc91684b5fbc4121e838436b37 P 5db6c31d6b604416b3ff9c95d1fa4931: stopping tablet replica
I20260925 02:53:18.762563 20732 raft_consensus.cc:2229] T 5a8a58dc91684b5fbc4121e838436b37 P 5db6c31d6b604416b3ff9c95d1fa4931 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:18.762619 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.762370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:39419 (local address 127.0.0.1:59986)
0925 02:53:18.762481 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.762549 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.193:39419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 02:53:18.762712 20732 raft_consensus.cc:2258] T 5a8a58dc91684b5fbc4121e838436b37 P 5db6c31d6b604416b3ff9c95d1fa4931 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 02:53:18.762782 19945 catalog_manager.cc:4584] TS 250e06c6d0614880b13576fa601a059f (127.12.222.193:39419): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a8a58dc91684b5fbc4121e838436b37: Network error: Client connection negotiation failed: client connection to 127.12.222.193:39419: connect: Connection refused (error 111)
I20260925 02:53:18.762972 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 20333
I20260925 02:53:18.769361 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 20470
I20260925 02:53:18.769541 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:18.769324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:43343 (local address 127.0.0.1:43278)
0925 02:53:18.769412 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:18.769465 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.194:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 02:53:18.776362 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 19814
I20260925 02:53:18.782305 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 19924
2026-09-25T02:53:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6839 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:53:18 dist-test-slave-252r krb5kdc[20736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:53:18 dist-test-slave-252r krb5kdc[20736](info): set up 2 sockets
Sep 25 02:53:18 dist-test-slave-252r krb5kdc[20736](info): commencing operation
krb5kdc: starting...
W20260925 02:53:19.740449 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:53:19 dist-test-slave-252r krb5kdc[20736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:53:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.254.keytab.
Entry for principal oryx/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.254.keytab.
I20260925 02:53:19.851166 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:44947
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.254.keytab
--principal=oryx/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:44947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:19.948345 20749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:19.948575 20749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:19.948668 20749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:19.952422 20749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:19.952488 20749 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:53:19.952518 20749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:19.952575 20749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:19.952610 20749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:19.952642 20749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:19.957211 20749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:44947
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.254.keytab
--principal=oryx/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:44947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:19.958009 20749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:19.958832 20749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:19.965793 20755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:19.965900 20758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:19.965909 20749 server_base.cc:1034] running on GCE node
W20260925 02:53:19.965814 20756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:19.966606 20749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:19.967487 20749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:19.968652 20749 hybrid_clock.cc:648] HybridClock initialized: now 1790304799968633 us; error 31 us; skew 500 ppm
Sep 25 02:53:19 dist-test-slave-252r krb5kdc[20736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304799, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:19.971035 20749 init.cc:377] Logged in from keytab as oryx/127.12.222.254@KRBTEST.COM (short username oryx)
I20260925 02:53:19.971794 20749 webserver.cc:460] Webserver started at http://127.12.222.254:37607/ using document root <none> and password file <none>
I20260925 02:53:19.972307 20749 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:19.972388 20749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:19.972560 20749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:19.974175 20749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "0424e79fe3ce45de984a88a5339e5da3"
format_stamp: "Formatted at 2026-09-25 02:53:19 on dist-test-slave-252r"
server_key: "44d46dd5090eb4d9b7c5461a5f8d4438"
server_key_iv: "6e934b78a1fe7be9b8122097ed6035a8"
server_key_version: "encryptionkey@0"
I20260925 02:53:19.974663 20749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "0424e79fe3ce45de984a88a5339e5da3"
format_stamp: "Formatted at 2026-09-25 02:53:19 on dist-test-slave-252r"
server_key: "44d46dd5090eb4d9b7c5461a5f8d4438"
server_key_iv: "6e934b78a1fe7be9b8122097ed6035a8"
server_key_version: "encryptionkey@0"
I20260925 02:53:19.977779 20749 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:53:19.979744 20766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:19.980733 20749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:19.980870 20749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "0424e79fe3ce45de984a88a5339e5da3"
format_stamp: "Formatted at 2026-09-25 02:53:19 on dist-test-slave-252r"
server_key: "44d46dd5090eb4d9b7c5461a5f8d4438"
server_key_iv: "6e934b78a1fe7be9b8122097ed6035a8"
server_key_version: "encryptionkey@0"
I20260925 02:53:19.980962 20749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:20.040987 20749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:20.043561 20749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:20.043745 20749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:20.050704 20749 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:44947
I20260925 02:53:20.050729 20829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:44947 every 8 connection(s)
I20260925 02:53:20.051543 20749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:20.053999 20830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:20.056867 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 20749
I20260925 02:53:20.057001 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:53:20.057307 13179 external_mini_cluster.cc:1428] Setting key 6efe47ff23249ef39def6c3075a76e12
I20260925 02:53:20.059561 20830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3: Bootstrap starting.
I20260925 02:53:20.061578 20830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:20.062233 20830 log.cc:824] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:20.063942 20830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3: No bootstrap required, opened a new log
Sep 25 02:53:20 dist-test-slave-252r krb5kdc[20736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM
I20260925 02:53:20.067678 20830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0424e79fe3ce45de984a88a5339e5da3" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 44947 } }
I20260925 02:53:20.067914 20830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:20.068009 20830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0424e79fe3ce45de984a88a5339e5da3, State: Initialized, Role: FOLLOWER
I20260925 02:53:20.068455 20830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0424e79fe3ce45de984a88a5339e5da3" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 44947 } }
I20260925 02:53:20.068599 20830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:20.068681 20830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:20.068776 20830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:20.069648 20830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0424e79fe3ce45de984a88a5339e5da3" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 44947 } }
I20260925 02:53:20.069967 20830 leader_election.cc:302] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0424e79fe3ce45de984a88a5339e5da3; no voters:
I20260925 02:53:20.070240 20830 leader_election.cc:288] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:53:20.070361 20835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:20.070425 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.058567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48988 (local address 127.12.222.254:44947)
0925 02:53:20.058938 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:20.058953 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:20.058984 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:53:20.059519 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:20.059533 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:20.059863 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:20.060157 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:20.060976 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.061891 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.062693 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.063107 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.063260 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.065814 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:20.065819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:20.065829 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:20.065853 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:20.067394 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:20.067872 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:20.067884 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:20.067888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:20.068102 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:20.068400 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:20.068404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:20.068405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:20.068850 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:20.069090 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:20.069329 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 02:53:20.069582 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20260925 02:53:20.070617 20835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3 [term 1 LEADER]: Becoming Leader. State: Replica: 0424e79fe3ce45de984a88a5339e5da3, State: Running, Role: LEADER
I20260925 02:53:20.070957 20835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0424e79fe3ce45de984a88a5339e5da3" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 44947 } }
I20260925 02:53:20.071399 20830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:20.073274 20837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0424e79fe3ce45de984a88a5339e5da3. Latest consensus state: current_term: 1 leader_uuid: "0424e79fe3ce45de984a88a5339e5da3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0424e79fe3ce45de984a88a5339e5da3" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 44947 } } }
I20260925 02:53:20.073406 20837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:20.074088 20836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0424e79fe3ce45de984a88a5339e5da3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0424e79fe3ce45de984a88a5339e5da3" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 44947 } } }
I20260925 02:53:20.074230 20836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3 [sys.catalog]: This master's current role is: LEADER
W20260925 02:53:20.076074 20850 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:53:20.076159 20850 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:53:20.076282 20851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:20.077137 20851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:20.082060 20851 catalog_manager.cc:1347] Generated new cluster ID: 7d5eab2ebd9d44e0b58dc687a5c777f9
I20260925 02:53:20.082131 20851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:20.101706 20851 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:20.102763 20851 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:20.107363 20851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3: Generated new TSK 0
I20260925 02:53:20.107944 20851 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.193.keytab.
Entry for principal oryx/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.193.keytab.
I20260925 02:53:20.135109 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:44947
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.193.keytab
--principal=oryx/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:20.237017 20857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:20.237247 20857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:20.237285 20857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:20.241096 20857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:20.241143 20857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:20.241179 20857 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:53:20.241214 20857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:53:20.246065 20857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.193.keytab
--principal=oryx/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.222.254:44947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:20.246852 20857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:20.247676 20857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:20.256400 20866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:20.256388 20864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:20.256400 20862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:20.256596 20857 server_base.cc:1034] running on GCE node
I20260925 02:53:20.257105 20857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:20.257616 20857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:20.258769 20857 hybrid_clock.cc:648] HybridClock initialized: now 1790304800258740 us; error 38 us; skew 500 ppm
Sep 25 02:53:20 dist-test-slave-252r krb5kdc[20736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304800, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:20.261399 20857 init.cc:377] Logged in from keytab as oryx/127.12.222.193@KRBTEST.COM (short username oryx)
I20260925 02:53:20.262183 20857 webserver.cc:460] Webserver started at http://127.12.222.193:34827/ using document root <none> and password file <none>
I20260925 02:53:20.262679 20857 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:20.262768 20857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:20.262980 20857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:20.264881 20857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "f970ce150ca54c86a02cf59183955b65"
format_stamp: "Formatted at 2026-09-25 02:53:20 on dist-test-slave-252r"
server_key: "f836ebbb7ca1ba92e918d0b594098309"
server_key_iv: "153d7453c6db300de9a671a70566ed76"
server_key_version: "encryptionkey@0"
I20260925 02:53:20.265497 20857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "f970ce150ca54c86a02cf59183955b65"
format_stamp: "Formatted at 2026-09-25 02:53:20 on dist-test-slave-252r"
server_key: "f836ebbb7ca1ba92e918d0b594098309"
server_key_iv: "153d7453c6db300de9a671a70566ed76"
server_key_version: "encryptionkey@0"
I20260925 02:53:20.268852 20857 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:53:20.271060 20873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:20.272292 20857 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 02:53:20.272424 20857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "f970ce150ca54c86a02cf59183955b65"
format_stamp: "Formatted at 2026-09-25 02:53:20 on dist-test-slave-252r"
server_key: "f836ebbb7ca1ba92e918d0b594098309"
server_key_iv: "153d7453c6db300de9a671a70566ed76"
server_key_version: "encryptionkey@0"
I20260925 02:53:20.272541 20857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:20.289510 20857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:20.291921 20857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:20.292126 20857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:20.293481 20857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:20.293529 20857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:20.293565 20857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:20.293578 20857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:53:20 dist-test-slave-252r krb5kdc[20736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304800, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM
I20260925 02:53:20.303897 20857 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:44517
I20260925 02:53:20.304059 20989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:44517 every 8 connection(s)
I20260925 02:53:20.304896 20857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:53:20.306895 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.294351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:51745 (local address 127.12.222.254:44947)
0925 02:53:20.294473 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:20.294476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:20.295501 (+ 1025us) server_negotiation.cc:407] Connection header received
0925 02:53:20.296617 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:20.296621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:20.296691 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:20.296792 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:20.297961 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.298789 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.299923 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.300357 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.300484 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.303287 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:20.303292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:20.303295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:20.303322 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:20.304739 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:20.305435 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:20.305439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:20.305441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:20.305533 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:20.305933 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:20.305937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:20.305939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:20.306161 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:20.306255 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:20.306673 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 02:53:20.306776 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:53:20.307550 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.294815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:44947 (local address 127.12.222.193:51745)
0925 02:53:20.295338 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:20.295404 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:20.296380 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:20.296967 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:20.296973 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:20.297357 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:20.297748 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.297786 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.298929 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.298933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.299817 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.299826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.300534 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.300538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.300768 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.301387 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:20.301402 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:20.303089 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:20.304849 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:20.304881 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:20.304894 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:20.305265 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:20.305659 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:20.305662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:20.305663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:20.305818 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:20.306269 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:20.306273 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:20.306471 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 02:53:20.306614 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 02:53:20.306823 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":152,"threads_started":1}
I20260925 02:53:20.308775 20990 heartbeater.cc:347] Connected to a master server at 127.12.222.254:44947
I20260925 02:53:20.308917 20990 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:20.309420 20990 heartbeater.cc:511] Master 127.12.222.254:44947 requested a full tablet report, sending...
I20260925 02:53:20.310453 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 20857
I20260925 02:53:20.310586 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:53:20.310928 13179 external_mini_cluster.cc:1428] Setting key d21cc191568b90b8c332fa9fbe23a923
I20260925 02:53:20.310968 20783 ts_manager.cc:194] Registered new tserver with Master: f970ce150ca54c86a02cf59183955b65 (127.12.222.193:44517)
I20260925 02:53:20.312466 20783 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:51745
I20260925 02:53:20.317889 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.310862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:58163 (local address 127.12.222.254:44947)
0925 02:53:20.310969 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:20.310973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:20.311244 (+ 271us) server_negotiation.cc:407] Connection header received
0925 02:53:20.311384 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:20.311389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:20.311445 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:20.311518 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:20.312052 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.312665 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.313565 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.313909 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.314010 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.315018 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:20.315021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:20.315024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:20.315046 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:20.316431 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:20.316806 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:20.316809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:20.316811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:20.316889 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:20.317147 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:20.317150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:20.317152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:20.317305 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:20.317426 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:20.317660 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 02:53:20.317756 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:53:20.317914 20996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.310816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:44947 (local address 127.12.222.193:58163)
0925 02:53:20.311163 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:20.311174 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:20.311281 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:20.311561 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:20.311563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:20.311777 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:20.311903 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.311911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.312777 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.312780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.313445 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.313455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.314016 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.314019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.314108 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.314574 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:20.314593 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:20.314859 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:20.316535 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:20.316538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:20.316540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:20.316714 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:20.316976 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:20.316979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:20.316981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:20.317063 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:20.317421 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:20.317424 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:20.317510 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 02:53:20.317609 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 02:53:20.317732 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for oryx/127.12.222.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.222.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.194.keytab.
Entry for principal oryx/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.194.keytab.
I20260925 02:53:20.336027 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:44947
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.194.keytab
--principal=oryx/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:20.435874 20999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:20.436076 20999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:20.436115 20999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:20.439637 20999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:20.439677 20999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:20.439724 20999 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:53:20.439765 20999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:53:20.444346 20999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.194.keytab
--principal=oryx/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.222.254:44947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:20.445097 20999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:20.445962 20999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:20.454245 21005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:20.454232 21004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:20.454232 21007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:20.454334 20999 server_base.cc:1034] running on GCE node
I20260925 02:53:20.454789 20999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:20.455368 20999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:20.456516 20999 hybrid_clock.cc:648] HybridClock initialized: now 1790304800456504 us; error 22 us; skew 500 ppm
Sep 25 02:53:20 dist-test-slave-252r krb5kdc[20736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304800, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:20.458868 20999 init.cc:377] Logged in from keytab as oryx/127.12.222.194@KRBTEST.COM (short username oryx)
I20260925 02:53:20.459511 20999 webserver.cc:460] Webserver started at http://127.12.222.194:39873/ using document root <none> and password file <none>
I20260925 02:53:20.459931 20999 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:20.459993 20999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:20.460136 20999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:20.461854 20999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "a9e65d464ca148ddada9f0d08ada07dd"
format_stamp: "Formatted at 2026-09-25 02:53:20 on dist-test-slave-252r"
server_key: "98d58695cdaea286c94862604ee779e3"
server_key_iv: "7f928f4234dee547a6310a038cbe1267"
server_key_version: "encryptionkey@0"
I20260925 02:53:20.462313 20999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "a9e65d464ca148ddada9f0d08ada07dd"
format_stamp: "Formatted at 2026-09-25 02:53:20 on dist-test-slave-252r"
server_key: "98d58695cdaea286c94862604ee779e3"
server_key_iv: "7f928f4234dee547a6310a038cbe1267"
server_key_version: "encryptionkey@0"
I20260925 02:53:20.465576 20999 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:53:20.467819 21014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:20.468993 20999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:20.469110 20999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "a9e65d464ca148ddada9f0d08ada07dd"
format_stamp: "Formatted at 2026-09-25 02:53:20 on dist-test-slave-252r"
server_key: "98d58695cdaea286c94862604ee779e3"
server_key_iv: "7f928f4234dee547a6310a038cbe1267"
server_key_version: "encryptionkey@0"
I20260925 02:53:20.469204 20999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:20.479629 20999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:20.482203 20999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:20.482393 20999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:20.483686 20999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:20.483753 20999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:20.483817 20999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:20.483857 20999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:53:20 dist-test-slave-252r krb5kdc[20736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304800, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM
I20260925 02:53:20.493882 20999 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:40261
I20260925 02:53:20.493968 21131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:40261 every 8 connection(s)
I20260925 02:53:20.494895 20999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:53:20.496424 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.484333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:55515 (local address 127.12.222.254:44947)
0925 02:53:20.484476 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:20.484480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:20.485335 (+ 855us) server_negotiation.cc:407] Connection header received
0925 02:53:20.486295 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:20.486301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:20.486363 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:20.486464 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:20.487773 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.488608 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.489696 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.489939 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.490055 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.492774 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:20.492777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:20.492780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:20.492810 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:20.494361 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:20.494813 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:20.494820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:20.494824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:20.494912 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:20.495289 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:20.495294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:20.495296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:20.495535 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:20.495630 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:20.496139 (+ 509us) server_negotiation.cc:299] Negotiation successful
0925 02:53:20.496275 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:53:20.497061 21023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.484735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:44947 (local address 127.12.222.194:55515)
0925 02:53:20.485160 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:20.485208 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:20.486051 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:20.486631 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:20.486639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:20.487081 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:20.487611 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.487632 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.488737 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.488742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.489568 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.489577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.490123 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.490131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.490353 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.490960 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:20.490982 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:20.492623 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:20.494473 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:20.494476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:20.494485 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:20.494709 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:20.495015 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:20.495018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:20.495020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:20.495164 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:20.495683 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:20.495688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:20.495931 (+ 243us) client_negotiation.cc:769] Sending connection context
0925 02:53:20.496077 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 02:53:20.496352 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":131,"threads_started":1}
I20260925 02:53:20.498137 21132 heartbeater.cc:347] Connected to a master server at 127.12.222.254:44947
I20260925 02:53:20.498282 21132 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:20.498739 21132 heartbeater.cc:511] Master 127.12.222.254:44947 requested a full tablet report, sending...
I20260925 02:53:20.499642 20783 ts_manager.cc:194] Registered new tserver with Master: a9e65d464ca148ddada9f0d08ada07dd (127.12.222.194:40261)
I20260925 02:53:20.500331 20783 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:55515
I20260925 02:53:20.501065 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 20999
I20260925 02:53:20.501187 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:53:20.501497 13179 external_mini_cluster.cc:1428] Setting key b2ffacbfe78488ace362484a64cd53c9
I20260925 02:53:20.506861 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.499949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:58721 (local address 127.12.222.254:44947)
0925 02:53:20.500048 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:20.500052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:20.500270 (+ 218us) server_negotiation.cc:407] Connection header received
0925 02:53:20.500411 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:20.500414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:20.500457 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:20.500516 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:20.500948 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.501411 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.502351 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.502622 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.502718 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.503689 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:20.503692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:20.503694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:20.503715 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:20.505322 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:20.505723 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:20.505726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:20.505728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:20.505815 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:20.506123 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:20.506125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:20.506126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:20.506307 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:20.506381 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:20.506681 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 02:53:20.506772 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:53:20.507033 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.499863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:44947 (local address 127.12.222.194:58721)
0925 02:53:20.500180 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:20.500190 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:20.500314 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:20.500530 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:20.500532 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:20.500711 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:20.500811 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.500836 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.501530 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.501533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.502234 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.502245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.502736 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.502740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.502829 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.503280 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:20.503290 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:20.503552 (+ 262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:20.505436 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:20.505445 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:20.505446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:20.505628 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:20.505916 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:20.505922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:20.505927 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:20.506035 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:20.506404 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:20.506410 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:20.506522 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 02:53:20.506633 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 02:53:20.506792 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for oryx/127.12.222.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.222.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.195.keytab.
Entry for principal oryx/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.195.keytab.
I20260925 02:53:20.527957 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:44947
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.195.keytab
--principal=oryx/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:20.628182 21142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:20.628479 21142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:20.628549 21142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:20.632007 21142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:20.632087 21142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:20.632154 21142 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:53:20.632220 21142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:53:20.636790 21142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.195.keytab
--principal=oryx/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.222.254:44947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:20.637653 21142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:20.638492 21142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:20.646814 21147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:20.646823 21148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:20.646956 21142 server_base.cc:1034] running on GCE node
W20260925 02:53:20.646783 21151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:20.647579 21142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:20.648118 21142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:20.649271 21142 hybrid_clock.cc:648] HybridClock initialized: now 1790304800649256 us; error 26 us; skew 500 ppm
Sep 25 02:53:20 dist-test-slave-252r krb5kdc[20736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304800, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:20.651855 21142 init.cc:377] Logged in from keytab as oryx/127.12.222.195@KRBTEST.COM (short username oryx)
I20260925 02:53:20.652648 21142 webserver.cc:460] Webserver started at http://127.12.222.195:34495/ using document root <none> and password file <none>
I20260925 02:53:20.653113 21142 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:20.653196 21142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:20.653396 21142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:20.655185 21142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a"
format_stamp: "Formatted at 2026-09-25 02:53:20 on dist-test-slave-252r"
server_key: "619da9a323d6b62fe340688cdea3e636"
server_key_iv: "ba3f9f0b8f283a1b54068debe7a0336f"
server_key_version: "encryptionkey@0"
I20260925 02:53:20.655704 21142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a"
format_stamp: "Formatted at 2026-09-25 02:53:20 on dist-test-slave-252r"
server_key: "619da9a323d6b62fe340688cdea3e636"
server_key_iv: "ba3f9f0b8f283a1b54068debe7a0336f"
server_key_version: "encryptionkey@0"
I20260925 02:53:20.659341 21142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 02:53:20.661465 21158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:20.662523 21142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:53:20.662662 21142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a"
format_stamp: "Formatted at 2026-09-25 02:53:20 on dist-test-slave-252r"
server_key: "619da9a323d6b62fe340688cdea3e636"
server_key_iv: "ba3f9f0b8f283a1b54068debe7a0336f"
server_key_version: "encryptionkey@0"
I20260925 02:53:20.662762 21142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:20.681176 21142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:20.683791 21142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:20.683964 21142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:20.685290 21142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:20.685364 21142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:20.685428 21142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:20.685479 21142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:53:20 dist-test-slave-252r krb5kdc[20736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304800, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM
I20260925 02:53:20.696408 21142 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:44815
I20260925 02:53:20.697203 21142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:53:20.697858 21275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:44815 every 8 connection(s)
I20260925 02:53:20.698818 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.687791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:49345 (local address 127.12.222.254:44947)
0925 02:53:20.687923 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:20.687927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:20.687942 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:53:20.688614 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:20.688618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:20.688686 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:20.688759 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:20.690133 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.690715 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.691786 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.692017 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.692128 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.695210 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:20.695214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:20.695216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:20.695246 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:20.696747 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:20.697215 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:20.697220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:20.697222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:20.697308 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:20.697697 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:20.697700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:20.697702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:20.697931 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:20.698016 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:20.698614 (+ 598us) server_negotiation.cc:299] Negotiation successful
0925 02:53:20.698708 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:53:20.699499 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.686759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:44947 (local address 127.12.222.195:49345)
0925 02:53:20.687298 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:20.687348 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:20.688362 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:20.689139 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:20.689148 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:20.689495 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:20.689989 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.690002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.690846 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.690849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.691651 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.691660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.692164 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.692168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.692384 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.693092 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:20.693114 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:20.695053 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:20.696862 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:20.696867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:20.696879 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:20.697111 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:20.697404 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:20.697408 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:20.697410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:20.697583 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:20.698062 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:20.698068 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:20.698391 (+ 323us) client_negotiation.cc:769] Sending connection context
0925 02:53:20.698542 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 02:53:20.698746 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":122,"threads_started":1}
I20260925 02:53:20.700614 21276 heartbeater.cc:347] Connected to a master server at 127.12.222.254:44947
I20260925 02:53:20.700734 21276 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:20.701220 21276 heartbeater.cc:511] Master 127.12.222.254:44947 requested a full tablet report, sending...
I20260925 02:53:20.702231 20783 ts_manager.cc:194] Registered new tserver with Master: f2d714ec2dab4e23bc6a4bb6e742ef9a (127.12.222.195:44815)
I20260925 02:53:20.702896 20783 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:49345
I20260925 02:53:20.702996 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 21142
I20260925 02:53:20.703125 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:53:20.703435 13179 external_mini_cluster.cc:1428] Setting key 4bb7838909fc9c05c96a42a6f489cc1c
I20260925 02:53:20.705420 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:53:20.709762 21282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.702665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:44947 (local address 127.12.222.195:53765)
0925 02:53:20.703044 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:20.703071 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:20.703162 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:20.703489 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:20.703492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:20.703669 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:20.703792 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.703806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.704652 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.704654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.705375 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.705386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.705820 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.705824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.705929 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.706340 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:20.706357 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:20.706757 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:20.708344 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:20.708347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:20.708349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:20.708584 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:20.708861 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:20.708864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:20.708865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:20.708940 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:20.709340 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:20.709341 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:20.709403 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 02:53:20.709503 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 02:53:20.709641 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":129,"threads_started":1}
I20260925 02:53:20.710071 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.702750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:53765 (local address 127.12.222.254:44947)
0925 02:53:20.702848 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:20.702852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:20.703127 (+ 275us) server_negotiation.cc:407] Connection header received
0925 02:53:20.703272 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:20.703275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:20.703320 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:20.703362 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:20.703906 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.704547 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.705504 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.705715 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.705790 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.706881 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:20.706884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:20.706885 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:20.706904 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:20.708196 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:20.708681 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:20.708687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:20.708691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:20.708771 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:20.709030 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:20.709035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:20.709040 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:20.709251 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:20.709345 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:20.709612 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 02:53:20.709726 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 02:53:20 dist-test-slave-252r krb5kdc[20736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304800, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:53:20 dist-test-slave-252r krb5kdc[20736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:20 dist-test-slave-252r krb5kdc[20736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:53:20.728051 20833 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:20.722648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49002 (local address 127.12.222.254:44947)
0925 02:53:20.722754 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:20.722757 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:20.722767 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:53:20.722793 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:20.722795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:20.722823 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:20.722900 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:20.723569 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.724036 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.724864 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.725103 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.725177 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.727913 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49002: BlockingRecv error: recv got EOF from 127.0.0.1:49002 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 02:53:20 dist-test-slave-252r krb5kdc[20736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304800, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM
I20260925 02:53:20.736030 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.729225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49014 (local address 127.12.222.254:44947)
0925 02:53:20.729333 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:20.729336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:20.729470 (+ 134us) server_negotiation.cc:407] Connection header received
0925 02:53:20.729551 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:20.729552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:20.729583 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:20.729642 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:20.730132 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.730561 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.731228 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.731418 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.731492 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.733610 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:20.733613 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:20.733614 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:20.733633 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:20.734737 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:20.735083 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:20.735085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:20.735086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:20.735150 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:20.735360 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:20.735362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:20.735363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:20.735509 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:20.735596 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:20.735771 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 02:53:20.735847 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:53:20.738576 20783 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:53:20.741420 20783 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:53:20.755398 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.749208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41098 (local address 127.12.222.195:44815)
0925 02:53:20.749526 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:20.749541 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:20.750592 (+ 1051us) server_negotiation.cc:407] Connection header received
0925 02:53:20.750808 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:20.750812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:20.750968 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:20.751081 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:20.751405 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.752120 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.753810 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.754509 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.754646 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.754760 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:20.755175 (+ 415us) server_negotiation.cc:299] Negotiation successful
0925 02:53:20.755260 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":102,"threads_started":1}
I20260925 02:53:20.755398 21303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.749894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41722 (local address 127.12.222.193:44517)
0925 02:53:20.750293 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:20.750304 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:20.750775 (+ 471us) server_negotiation.cc:407] Connection header received
0925 02:53:20.750856 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:20.750860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:20.751002 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:20.751173 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:20.751457 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.752605 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.754047 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.754791 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.754927 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.755053 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:20.755175 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 02:53:20.755263 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":108,"threads_started":1}
I20260925 02:53:20.755429 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.749720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:44517 (local address 127.0.0.1:41722)
0925 02:53:20.750478 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:20.750508 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:20.750641 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:20.751163 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:20.751166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:20.751196 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:20.751310 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.751317 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.752736 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.752740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.753924 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.753934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.754919 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.754933 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.755034 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.755055 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 02:53:20.755141 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 02:53:20.755281 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":4,"spinlock_wait_cycles":768,"thread_start_us":81,"threads_started":1}
I20260925 02:53:20.755833 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.749442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:40261 (local address 127.0.0.1:48212)
0925 02:53:20.750439 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:20.750509 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:20.750644 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:20.751734 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:20.751737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:20.751754 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:20.751859 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.751864 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.753246 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.753249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.754219 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.754225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.755193 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.755196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.755318 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.755338 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 02:53:20.755405 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 02:53:20.755701 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":29,"spinlock_wait_cycles":3328,"thread_start_us":93,"threads_started":1}
I20260925 02:53:20.756088 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.749024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:44815 (local address 127.0.0.1:41098)
0925 02:53:20.750439 (+ 1415us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:20.750508 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:20.750652 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:20.751124 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:20.751129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:20.751151 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:20.751284 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.751290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.752271 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.752275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.753681 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.753691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.754828 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.754834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.754946 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.754982 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 02:53:20.755565 (+ 583us) client_negotiation.cc:240] Negotiation successful
0925 02:53:20.755921 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":233,"spinlock_wait_cycles":1792,"thread_start_us":74,"threads_started":1}
I20260925 02:53:20.756325 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.749849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.12.222.194:40261)
0925 02:53:20.750257 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:20.750273 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:20.750603 (+ 330us) server_negotiation.cc:407] Connection header received
0925 02:53:20.751281 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:20.751288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:20.751465 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:20.751605 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:20.751990 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.753122 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.754325 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.754876 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.755010 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.755134 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:20.756071 (+ 937us) server_negotiation.cc:299] Negotiation successful
0925 02:53:20.756155 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":98,"threads_started":1}
I20260925 02:53:20.758186 20923 tablet_service.cc:1467] Processing CreateTablet for tablet e9b587141b6c4bc3b6bfdf45b026de7a (DEFAULT_TABLE table=test-table [id=47d0261293ff4ca4834eccaddf31826b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:20.759040 20923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b587141b6c4bc3b6bfdf45b026de7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:20.758836 21209 tablet_service.cc:1467] Processing CreateTablet for tablet e9b587141b6c4bc3b6bfdf45b026de7a (DEFAULT_TABLE table=test-table [id=47d0261293ff4ca4834eccaddf31826b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:20.758836 21064 tablet_service.cc:1467] Processing CreateTablet for tablet e9b587141b6c4bc3b6bfdf45b026de7a (DEFAULT_TABLE table=test-table [id=47d0261293ff4ca4834eccaddf31826b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:20.759733 21064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b587141b6c4bc3b6bfdf45b026de7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:20.759738 21209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b587141b6c4bc3b6bfdf45b026de7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:20.765550 21304 tablet_bootstrap.cc:492] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65: Bootstrap starting.
I20260925 02:53:20.766731 21305 tablet_bootstrap.cc:492] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a: Bootstrap starting.
I20260925 02:53:20.767725 21306 tablet_bootstrap.cc:492] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd: Bootstrap starting.
I20260925 02:53:20.767926 21304 tablet_bootstrap.cc:654] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:20.768805 21305 tablet_bootstrap.cc:654] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:20.768878 21304 log.cc:824] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:20.769521 21305 log.cc:824] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:20.769788 21306 tablet_bootstrap.cc:654] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:20.770390 21306 log.cc:824] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:20.770654 21304 tablet_bootstrap.cc:492] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65: No bootstrap required, opened a new log
I20260925 02:53:20.770848 21304 ts_tablet_manager.cc:1397] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:53:20.772536 21306 tablet_bootstrap.cc:492] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd: No bootstrap required, opened a new log
I20260925 02:53:20.772708 21306 ts_tablet_manager.cc:1397] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:53:20.774641 21304 raft_consensus.cc:348] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } }
I20260925 02:53:20.774894 21304 raft_consensus.cc:374] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:20.774961 21304 raft_consensus.cc:729] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f970ce150ca54c86a02cf59183955b65, State: Initialized, Role: FOLLOWER
I20260925 02:53:20.775395 21304 consensus_queue.cc:260] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } }
I20260925 02:53:20.776049 20990 heartbeater.cc:503] Master 127.12.222.254:44947 was elected leader, sending a full tablet report...
I20260925 02:53:20.776346 21304 ts_tablet_manager.cc:1428] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 02:53:20.776384 21306 raft_consensus.cc:348] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } }
I20260925 02:53:20.776654 21306 raft_consensus.cc:374] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:20.776733 21306 raft_consensus.cc:729] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e65d464ca148ddada9f0d08ada07dd, State: Initialized, Role: FOLLOWER
I20260925 02:53:20.777150 21306 consensus_queue.cc:260] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } }
I20260925 02:53:20.777896 21305 tablet_bootstrap.cc:492] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a: No bootstrap required, opened a new log
I20260925 02:53:20.778024 21306 ts_tablet_manager.cc:1428] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 02:53:20.778062 21305 ts_tablet_manager.cc:1397] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20260925 02:53:20.778842 21132 heartbeater.cc:503] Master 127.12.222.254:44947 was elected leader, sending a full tablet report...
I20260925 02:53:20.780910 21305 raft_consensus.cc:348] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } }
I20260925 02:53:20.781095 21305 raft_consensus.cc:374] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:20.781162 21305 raft_consensus.cc:729] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d714ec2dab4e23bc6a4bb6e742ef9a, State: Initialized, Role: FOLLOWER
I20260925 02:53:20.781489 21305 consensus_queue.cc:260] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } }
I20260925 02:53:20.782028 21276 heartbeater.cc:503] Master 127.12.222.254:44947 was elected leader, sending a full tablet report...
I20260925 02:53:20.782297 21305 ts_tablet_manager.cc:1428] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260925 02:53:20.789601 21310 raft_consensus.cc:482] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:20.789722 21310 raft_consensus.cc:504] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } }
I20260925 02:53:20.790500 21310 leader_election.cc:288] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2d714ec2dab4e23bc6a4bb6e742ef9a (127.12.222.195:44815), a9e65d464ca148ddada9f0d08ada07dd (127.12.222.194:40261)
I20260925 02:53:20.793762 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.790813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:60593 (local address 127.12.222.195:44815)
0925 02:53:20.790949 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:20.790952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:20.790963 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:53:20.790996 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:20.790998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:20.791028 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:20.791110 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:20.791334 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.791830 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.792858 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.793233 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.793352 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.793401 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:20.793586 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 02:53:20.793645 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:53:20.793776 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.790648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:44815 (local address 127.12.222.193:60593)
0925 02:53:20.790744 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:20.790758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:20.790833 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:20.791120 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:20.791123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:20.791139 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:20.791237 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.791241 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.791927 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.791932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.792728 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.792737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.793345 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.793349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.793468 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.793485 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 02:53:20.793570 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 02:53:20.793632 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 02:53:20.793990 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.790798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:48393 (local address 127.12.222.194:40261)
0925 02:53:20.790906 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:20.790913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:20.791086 (+ 173us) server_negotiation.cc:407] Connection header received
0925 02:53:20.791207 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:20.791211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:20.791251 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:20.791324 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:20.791619 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.792089 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.793099 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.793481 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.793598 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.793642 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:20.793833 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 02:53:20.793893 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:53:20.794003 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.790709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:40261 (local address 127.12.222.193:48393)
0925 02:53:20.791013 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:20.791022 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:20.791115 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:20.791335 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:20.791338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:20.791352 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:20.791512 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.791517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.792196 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.792199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.792980 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:20.792988 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.793595 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.793598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:20.793698 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.793726 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 02:53:20.793818 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 02:53:20.793879 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":99,"threads_started":1}
I20260925 02:53:20.794397 21229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b587141b6c4bc3b6bfdf45b026de7a" candidate_uuid: "f970ce150ca54c86a02cf59183955b65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" is_pre_election: true
I20260925 02:53:20.794549 21084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b587141b6c4bc3b6bfdf45b026de7a" candidate_uuid: "f970ce150ca54c86a02cf59183955b65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e65d464ca148ddada9f0d08ada07dd" is_pre_election: true
I20260925 02:53:20.794628 21229 raft_consensus.cc:2454] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f970ce150ca54c86a02cf59183955b65 in term 0.
I20260925 02:53:20.794797 21084 raft_consensus.cc:2454] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f970ce150ca54c86a02cf59183955b65 in term 0.
I20260925 02:53:20.795163 20877 leader_election.cc:302] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f2d714ec2dab4e23bc6a4bb6e742ef9a, f970ce150ca54c86a02cf59183955b65; no voters:
I20260925 02:53:20.795384 21310 raft_consensus.cc:2781] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:53:20.795464 21310 raft_consensus.cc:482] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:53:20.795508 21310 raft_consensus.cc:3037] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:20.796643 21310 raft_consensus.cc:504] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } }
I20260925 02:53:20.796962 21310 leader_election.cc:288] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [CANDIDATE]: Term 1 election: Requested vote from peers f2d714ec2dab4e23bc6a4bb6e742ef9a (127.12.222.195:44815), a9e65d464ca148ddada9f0d08ada07dd (127.12.222.194:40261)
I20260925 02:53:20.797222 21229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b587141b6c4bc3b6bfdf45b026de7a" candidate_uuid: "f970ce150ca54c86a02cf59183955b65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a"
I20260925 02:53:20.797338 21229 raft_consensus.cc:3037] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:20.797365 21084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b587141b6c4bc3b6bfdf45b026de7a" candidate_uuid: "f970ce150ca54c86a02cf59183955b65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e65d464ca148ddada9f0d08ada07dd"
I20260925 02:53:20.797478 21084 raft_consensus.cc:3037] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:20.798362 21229 raft_consensus.cc:2454] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f970ce150ca54c86a02cf59183955b65 in term 1.
I20260925 02:53:20.798485 21084 raft_consensus.cc:2454] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f970ce150ca54c86a02cf59183955b65 in term 1.
I20260925 02:53:20.798702 20877 leader_election.cc:302] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f2d714ec2dab4e23bc6a4bb6e742ef9a, f970ce150ca54c86a02cf59183955b65; no voters:
I20260925 02:53:20.798899 21310 raft_consensus.cc:2781] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:20.799049 21310 raft_consensus.cc:686] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [term 1 LEADER]: Becoming Leader. State: Replica: f970ce150ca54c86a02cf59183955b65, State: Running, Role: LEADER
I20260925 02:53:20.799307 21310 consensus_queue.cc:237] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } }
I20260925 02:53:20.802147 20784 catalog_manager.cc:5495] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f970ce150ca54c86a02cf59183955b65 (127.12.222.193). New cstate: current_term: 1 leader_uuid: "f970ce150ca54c86a02cf59183955b65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } health_report { overall_health: UNKNOWN } } }
W20260925 02:53:20.806083 20991 tablet.cc:2369] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:53:20.815757 21303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:20.812481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.12.222.193:44517)
0925 02:53:20.812586 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:20.812592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:20.812660 (+ 68us) server_negotiation.cc:407] Connection header received
0925 02:53:20.812846 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:20.812854 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:20.812899 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:20.813000 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:53:20.813228 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.813677 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.814449 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:20.814642 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:20.814736 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:20.815019 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:53:20.815382 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:53:20.815504 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:20.815588 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 02:53:20.815634 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:53:20.820573 21084 raft_consensus.cc:1261] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd [term 1 FOLLOWER]: Refusing update from remote peer f970ce150ca54c86a02cf59183955b65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:20.820873 21229 raft_consensus.cc:1261] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 FOLLOWER]: Refusing update from remote peer f970ce150ca54c86a02cf59183955b65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:20.821233 21314 consensus_queue.cc:1035] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:20.821381 21310 consensus_queue.cc:1035] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:20.826756 21317 mvcc.cc:205] Tried to move back new op lower bound from 7333088464155283456 to 7333088464076464128. Current Snapshot: MvccSnapshot[applied={T|T < 7333088464155283456}]
I20260925 02:53:20.828101 21319 mvcc.cc:205] Tried to move back new op lower bound from 7333088464155283456 to 7333088464076464128. Current Snapshot: MvccSnapshot[applied={T|T < 7333088464155283456}]
I20260925 02:53:20.829527 21318 mvcc.cc:205] Tried to move back new op lower bound from 7333088464155283456 to 7333088464076464128. Current Snapshot: MvccSnapshot[applied={T|T < 7333088464155283456}]
I20260925 02:53:20.835484 20784 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:20.835638 20784 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:20.837792 20784 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3: Sending DeleteTablet for 3 replicas of tablet e9b587141b6c4bc3b6bfdf45b026de7a
I20260925 02:53:20.838560 21064 tablet_service.cc:1514] Processing DeleteTablet for tablet e9b587141b6c4bc3b6bfdf45b026de7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:20 UTC) from {username='oryx', principal='oryx/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:48212
I20260925 02:53:20.838650 20923 tablet_service.cc:1514] Processing DeleteTablet for tablet e9b587141b6c4bc3b6bfdf45b026de7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:20 UTC) from {username='oryx', principal='oryx/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:41722
I20260925 02:53:20.838650 21209 tablet_service.cc:1514] Processing DeleteTablet for tablet e9b587141b6c4bc3b6bfdf45b026de7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:20 UTC) from {username='oryx', principal='oryx/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:41098
I20260925 02:53:20.839103 21327 tablet_replica.cc:330] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd: stopping tablet replica
I20260925 02:53:20.839355 21329 tablet_replica.cc:330] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a: stopping tablet replica
I20260925 02:53:20.839406 21327 raft_consensus.cc:2229] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:20.839617 21329 raft_consensus.cc:2229] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:20.839562 20784 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 02:53:20.839653 21327 raft_consensus.cc:2258] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:20.839847 21329 raft_consensus.cc:2258] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 02:53:20.839928 20784 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:53:20.840662 21328 tablet_replica.cc:330] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65: stopping tablet replica
I20260925 02:53:20.840984 21328 raft_consensus.cc:2229] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:53:20.841385 21328 raft_consensus.cc:2258] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:20.841388 21329 ts_tablet_manager.cc:1905] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:20.843948 21327 ts_tablet_manager.cc:1905] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:20.844158 21328 ts_tablet_manager.cc:1905] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:20.844431 21329 ts_tablet_manager.cc:1918] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:20.844524 21329 log.cc:1196] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/e9b587141b6c4bc3b6bfdf45b026de7a
I20260925 02:53:20.844732 21064 tablet_service.cc:1467] Processing CreateTablet for tablet a258fd63d7374233805215f8d9eaa04c (DEFAULT_TABLE table=test-table [id=c4646ade5a154c49875b560335354111]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:20.844808 21329 ts_tablet_manager.cc:1939] T e9b587141b6c4bc3b6bfdf45b026de7a P f2d714ec2dab4e23bc6a4bb6e742ef9a: Deleting consensus metadata
I20260925 02:53:20.844985 21064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a258fd63d7374233805215f8d9eaa04c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:20.846594 21327 ts_tablet_manager.cc:1918] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:20.846698 21327 log.cc:1196] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/e9b587141b6c4bc3b6bfdf45b026de7a
I20260925 02:53:20.846968 21327 ts_tablet_manager.cc:1939] T e9b587141b6c4bc3b6bfdf45b026de7a P a9e65d464ca148ddada9f0d08ada07dd: Deleting consensus metadata
I20260925 02:53:20.847146 21306 tablet_bootstrap.cc:492] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd: Bootstrap starting.
I20260925 02:53:20.847281 21328 ts_tablet_manager.cc:1918] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:20.847361 21328 log.cc:1196] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/wal/wals/e9b587141b6c4bc3b6bfdf45b026de7a
I20260925 02:53:20.847479 20770 catalog_manager.cc:4841] TS f2d714ec2dab4e23bc6a4bb6e742ef9a (127.12.222.195:44815): tablet e9b587141b6c4bc3b6bfdf45b026de7a (table test-table [id=47d0261293ff4ca4834eccaddf31826b]) successfully deleted
I20260925 02:53:20.847689 21328 ts_tablet_manager.cc:1939] T e9b587141b6c4bc3b6bfdf45b026de7a P f970ce150ca54c86a02cf59183955b65: Deleting consensus metadata
I20260925 02:53:20.847970 21306 tablet_bootstrap.cc:654] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:20.847934 20768 catalog_manager.cc:4841] TS a9e65d464ca148ddada9f0d08ada07dd (127.12.222.194:40261): tablet e9b587141b6c4bc3b6bfdf45b026de7a (table test-table [id=47d0261293ff4ca4834eccaddf31826b]) successfully deleted
I20260925 02:53:20.847982 20923 tablet_service.cc:1467] Processing CreateTablet for tablet a258fd63d7374233805215f8d9eaa04c (DEFAULT_TABLE table=test-table [id=c4646ade5a154c49875b560335354111]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:20.848181 20923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a258fd63d7374233805215f8d9eaa04c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:20.848583 20770 catalog_manager.cc:4841] TS f970ce150ca54c86a02cf59183955b65 (127.12.222.193:44517): tablet e9b587141b6c4bc3b6bfdf45b026de7a (table test-table [id=47d0261293ff4ca4834eccaddf31826b]) successfully deleted
I20260925 02:53:20.849167 21306 tablet_bootstrap.cc:492] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd: No bootstrap required, opened a new log
I20260925 02:53:20.849242 21306 ts_tablet_manager.cc:1397] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:20.849658 21306 raft_consensus.cc:348] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:20.849700 21209 tablet_service.cc:1467] Processing CreateTablet for tablet a258fd63d7374233805215f8d9eaa04c (DEFAULT_TABLE table=test-table [id=c4646ade5a154c49875b560335354111]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:20.849752 21306 raft_consensus.cc:374] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:20.849781 21306 raft_consensus.cc:729] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e65d464ca148ddada9f0d08ada07dd, State: Initialized, Role: FOLLOWER
I20260925 02:53:20.849920 21209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a258fd63d7374233805215f8d9eaa04c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:20.849885 21306 consensus_queue.cc:260] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:20.850118 21306 ts_tablet_manager.cc:1428] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:53:20.850268 21304 tablet_bootstrap.cc:492] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65: Bootstrap starting.
I20260925 02:53:20.851159 21304 tablet_bootstrap.cc:654] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:20.852370 21304 tablet_bootstrap.cc:492] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65: No bootstrap required, opened a new log
I20260925 02:53:20.852195 21305 tablet_bootstrap.cc:492] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a: Bootstrap starting.
I20260925 02:53:20.852447 21304 ts_tablet_manager.cc:1397] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:20.852892 21304 raft_consensus.cc:348] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:20.853008 21304 raft_consensus.cc:374] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:20.853044 21304 raft_consensus.cc:729] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f970ce150ca54c86a02cf59183955b65, State: Initialized, Role: FOLLOWER
I20260925 02:53:20.853145 21304 consensus_queue.cc:260] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:20.853255 21305 tablet_bootstrap.cc:654] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:20.853406 21304 ts_tablet_manager.cc:1428] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:53:20.854362 21305 tablet_bootstrap.cc:492] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a: No bootstrap required, opened a new log
I20260925 02:53:20.854450 21305 ts_tablet_manager.cc:1397] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 02:53:20.854904 21305 raft_consensus.cc:348] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:20.855032 21305 raft_consensus.cc:374] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:20.855098 21305 raft_consensus.cc:729] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d714ec2dab4e23bc6a4bb6e742ef9a, State: Initialized, Role: FOLLOWER
I20260925 02:53:20.855225 21305 consensus_queue.cc:260] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:20.855492 21305 ts_tablet_manager.cc:1428] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260925 02:53:20.948109 21277 tablet.cc:2369] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:53:20.997037 21133 tablet.cc:2369] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:53:21.237612 21324 raft_consensus.cc:482] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:21.237739 21324 raft_consensus.cc:504] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.238081 21324 leader_election.cc:288] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9e65d464ca148ddada9f0d08ada07dd (127.12.222.194:40261), f2d714ec2dab4e23bc6a4bb6e742ef9a (127.12.222.195:44815)
I20260925 02:53:21.238555 21084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a258fd63d7374233805215f8d9eaa04c" candidate_uuid: "f970ce150ca54c86a02cf59183955b65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e65d464ca148ddada9f0d08ada07dd" is_pre_election: true
I20260925 02:53:21.238554 21229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a258fd63d7374233805215f8d9eaa04c" candidate_uuid: "f970ce150ca54c86a02cf59183955b65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" is_pre_election: true
I20260925 02:53:21.238716 21229 raft_consensus.cc:2454] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f970ce150ca54c86a02cf59183955b65 in term 0.
I20260925 02:53:21.238716 21084 raft_consensus.cc:2454] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f970ce150ca54c86a02cf59183955b65 in term 0.
I20260925 02:53:21.239033 20877 leader_election.cc:302] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f2d714ec2dab4e23bc6a4bb6e742ef9a, f970ce150ca54c86a02cf59183955b65; no voters:
I20260925 02:53:21.239195 21324 raft_consensus.cc:2781] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:53:21.239254 21324 raft_consensus.cc:482] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:53:21.239296 21324 raft_consensus.cc:3037] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:21.240320 21324 raft_consensus.cc:504] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.240593 21324 leader_election.cc:288] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [CANDIDATE]: Term 1 election: Requested vote from peers a9e65d464ca148ddada9f0d08ada07dd (127.12.222.194:40261), f2d714ec2dab4e23bc6a4bb6e742ef9a (127.12.222.195:44815)
I20260925 02:53:21.240900 21084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a258fd63d7374233805215f8d9eaa04c" candidate_uuid: "f970ce150ca54c86a02cf59183955b65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e65d464ca148ddada9f0d08ada07dd"
I20260925 02:53:21.240900 21229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a258fd63d7374233805215f8d9eaa04c" candidate_uuid: "f970ce150ca54c86a02cf59183955b65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a"
I20260925 02:53:21.241035 21084 raft_consensus.cc:3037] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:21.241035 21229 raft_consensus.cc:3037] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:21.242161 21084 raft_consensus.cc:2454] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f970ce150ca54c86a02cf59183955b65 in term 1.
I20260925 02:53:21.242161 21229 raft_consensus.cc:2454] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f970ce150ca54c86a02cf59183955b65 in term 1.
I20260925 02:53:21.242461 20875 leader_election.cc:302] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9e65d464ca148ddada9f0d08ada07dd, f970ce150ca54c86a02cf59183955b65; no voters:
I20260925 02:53:21.242604 21324 raft_consensus.cc:2781] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:21.242717 21324 raft_consensus.cc:686] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [term 1 LEADER]: Becoming Leader. State: Replica: f970ce150ca54c86a02cf59183955b65, State: Running, Role: LEADER
I20260925 02:53:21.242846 21324 consensus_queue.cc:237] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.244496 20784 catalog_manager.cc:5495] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f970ce150ca54c86a02cf59183955b65 (127.12.222.193). New cstate: current_term: 1 leader_uuid: "f970ce150ca54c86a02cf59183955b65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } health_report { overall_health: UNKNOWN } } }
Sep 25 02:53:21 dist-test-slave-252r krb5kdc[20736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304799, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.254@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM
I20260925 02:53:21.330147 21340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.321827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49030 (local address 127.12.222.254:44947)
0925 02:53:21.322062 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:21.322070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:21.322195 (+ 125us) server_negotiation.cc:407] Connection header received
0925 02:53:21.322344 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:21.322349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:21.322389 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:21.322486 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:21.323114 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.323633 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.324466 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.324685 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.324792 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.327012 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:21.327018 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:21.327024 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:21.327058 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:21.328496 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:21.328879 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:21.328881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:21.328882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:21.328967 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:21.329297 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:21.329299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:21.329301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:21.329471 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:21.329576 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:21.329849 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 02:53:21.330011 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":173,"thread_start_us":96,"threads_started":1}
I20260925 02:53:21.330147 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.321786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:44947 (local address 127.0.0.1:49030)
0925 02:53:21.322086 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:21.322108 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:21.322252 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:21.322503 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:21.322505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:21.322897 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:21.322994 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.323001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.323734 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.323739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.324309 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.324318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.324791 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.324794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.324863 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.325484 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:21.325496 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:21.326872 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:21.328585 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:21.328592 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:21.328597 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:21.328800 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:21.329053 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:21.329056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:21.329058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:21.329159 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:21.329574 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:21.329581 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:21.329686 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 02:53:21.329795 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 02:53:21.329947 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
I20260925 02:53:21.333710 20784 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:49030:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:53:21.334297 20784 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:53:21.338762 21064 tablet_service.cc:1467] Processing CreateTablet for tablet 032c351ad97a404a94e3f7b038bf38c7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=98668c06b6084039ac04c21aaf6908e9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:53:21.338762 20923 tablet_service.cc:1467] Processing CreateTablet for tablet 032c351ad97a404a94e3f7b038bf38c7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=98668c06b6084039ac04c21aaf6908e9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:53:21.339126 21064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 032c351ad97a404a94e3f7b038bf38c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:21.339126 20923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 032c351ad97a404a94e3f7b038bf38c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:21.339066 21209 tablet_service.cc:1467] Processing CreateTablet for tablet 032c351ad97a404a94e3f7b038bf38c7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=98668c06b6084039ac04c21aaf6908e9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:53:21.339346 21209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 032c351ad97a404a94e3f7b038bf38c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:21.341624 21305 tablet_bootstrap.cc:492] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a: Bootstrap starting.
I20260925 02:53:21.341774 21304 tablet_bootstrap.cc:492] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65: Bootstrap starting.
I20260925 02:53:21.342386 21306 tablet_bootstrap.cc:492] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd: Bootstrap starting.
I20260925 02:53:21.342537 21305 tablet_bootstrap.cc:654] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:21.342623 21304 tablet_bootstrap.cc:654] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:21.343272 21306 tablet_bootstrap.cc:654] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:21.343696 21304 tablet_bootstrap.cc:492] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65: No bootstrap required, opened a new log
I20260925 02:53:21.343787 21304 ts_tablet_manager.cc:1397] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:53:21.344209 21304 raft_consensus.cc:348] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.344345 21304 raft_consensus.cc:374] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:21.344385 21304 raft_consensus.cc:729] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f970ce150ca54c86a02cf59183955b65, State: Initialized, Role: FOLLOWER
I20260925 02:53:21.344488 21306 tablet_bootstrap.cc:492] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd: No bootstrap required, opened a new log
I20260925 02:53:21.344501 21304 consensus_queue.cc:260] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.344566 21306 ts_tablet_manager.cc:1397] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:21.344764 21304 ts_tablet_manager.cc:1428] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:53:21.344955 21306 raft_consensus.cc:348] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.344954 21324 tablet_replica.cc:439] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } }
I20260925 02:53:21.345078 21306 raft_consensus.cc:374] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:21.345098 21324 tablet_replica.cc:442] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:53:21.345126 21306 raft_consensus.cc:729] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e65d464ca148ddada9f0d08ada07dd, State: Initialized, Role: FOLLOWER
I20260925 02:53:21.345248 21306 consensus_queue.cc:260] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.345504 21306 ts_tablet_manager.cc:1428] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:53:21.345638 21311 tablet_replica.cc:439] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } }
I20260925 02:53:21.345777 21311 tablet_replica.cc:442] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:53:21.346632 21305 tablet_bootstrap.cc:492] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a: No bootstrap required, opened a new log
I20260925 02:53:21.346705 21305 ts_tablet_manager.cc:1397] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 02:53:21.347071 21305 raft_consensus.cc:348] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.347205 21305 raft_consensus.cc:374] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:21.347239 21305 raft_consensus.cc:729] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d714ec2dab4e23bc6a4bb6e742ef9a, State: Initialized, Role: FOLLOWER
I20260925 02:53:21.347357 21305 consensus_queue.cc:260] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.347658 21305 ts_tablet_manager.cc:1428] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:53:21.347600 21312 tablet_replica.cc:439] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } }
I20260925 02:53:21.347749 21312 tablet_replica.cc:442] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:53:21.431839 21311 raft_consensus.cc:482] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:21.431959 21311 raft_consensus.cc:504] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.432966 21311 leader_election.cc:288] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f970ce150ca54c86a02cf59183955b65 (127.12.222.193:44517), f2d714ec2dab4e23bc6a4bb6e742ef9a (127.12.222.195:44815)
I20260925 02:53:21.437455 21345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.433268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:44823 (local address 127.12.222.193:44517)
0925 02:53:21.433493 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:21.433497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:21.433537 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:53:21.433589 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:21.433592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:21.433637 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:21.433713 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:21.433994 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.434695 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.436321 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.436883 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.437003 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.437054 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:21.437258 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 02:53:21.437327 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":73,"threads_started":1}
I20260925 02:53:21.437456 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.433600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:44869 (local address 127.12.222.195:44815)
0925 02:53:21.433829 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:21.433832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:21.433853 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:53:21.433888 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:21.433890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:21.433942 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:21.434023 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:21.434229 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.434870 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.436320 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.436887 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.437014 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.437054 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:21.437258 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 02:53:21.437327 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":77,"threads_started":1}
I20260925 02:53:21.437526 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.433508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:44815 (local address 127.12.222.194:44869)
0925 02:53:21.433740 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:21.433751 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:21.433819 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:21.434037 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:21.434039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:21.434054 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:21.434132 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.434136 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.434969 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.434971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.436161 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.436173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.437010 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.437013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.437136 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.437154 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 02:53:21.437229 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:53:21.437332 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":94,"spinlock_wait_cycles":1792,"thread_start_us":72,"threads_started":1}
I20260925 02:53:21.437698 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.433169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:44517 (local address 127.12.222.194:44823)
0925 02:53:21.433382 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:21.433398 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:21.433481 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:21.433739 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:21.433741 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:21.433764 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:21.433881 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.433887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.434802 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.434807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.436178 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.436184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.437010 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.437013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.437125 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.437153 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 02:53:21.437229 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 02:53:21.437332 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":74,"thread_start_us":74,"threads_started":1}
I20260925 02:53:21.437929 21229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032c351ad97a404a94e3f7b038bf38c7" candidate_uuid: "a9e65d464ca148ddada9f0d08ada07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" is_pre_election: true
I20260925 02:53:21.438071 21229 raft_consensus.cc:2454] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9e65d464ca148ddada9f0d08ada07dd in term 0.
I20260925 02:53:21.438292 20943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032c351ad97a404a94e3f7b038bf38c7" candidate_uuid: "a9e65d464ca148ddada9f0d08ada07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f970ce150ca54c86a02cf59183955b65" is_pre_election: true
I20260925 02:53:21.438416 21018 leader_election.cc:302] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9e65d464ca148ddada9f0d08ada07dd, f2d714ec2dab4e23bc6a4bb6e742ef9a; no voters:
I20260925 02:53:21.438529 20943 raft_consensus.cc:2454] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9e65d464ca148ddada9f0d08ada07dd in term 0.
I20260925 02:53:21.438699 21311 raft_consensus.cc:2781] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:53:21.438760 21311 raft_consensus.cc:482] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:53:21.438817 21311 raft_consensus.cc:3037] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:21.439781 21311 raft_consensus.cc:504] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.440104 21311 leader_election.cc:288] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [CANDIDATE]: Term 1 election: Requested vote from peers f970ce150ca54c86a02cf59183955b65 (127.12.222.193:44517), f2d714ec2dab4e23bc6a4bb6e742ef9a (127.12.222.195:44815)
I20260925 02:53:21.440460 20943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032c351ad97a404a94e3f7b038bf38c7" candidate_uuid: "a9e65d464ca148ddada9f0d08ada07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f970ce150ca54c86a02cf59183955b65"
I20260925 02:53:21.440500 21229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032c351ad97a404a94e3f7b038bf38c7" candidate_uuid: "a9e65d464ca148ddada9f0d08ada07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a"
I20260925 02:53:21.440557 20943 raft_consensus.cc:3037] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:21.440634 21229 raft_consensus.cc:3037] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:21.441450 21229 raft_consensus.cc:2454] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9e65d464ca148ddada9f0d08ada07dd in term 1.
I20260925 02:53:21.441510 20943 raft_consensus.cc:2454] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9e65d464ca148ddada9f0d08ada07dd in term 1.
I20260925 02:53:21.441720 21018 leader_election.cc:302] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9e65d464ca148ddada9f0d08ada07dd, f2d714ec2dab4e23bc6a4bb6e742ef9a; no voters:
I20260925 02:53:21.441881 21311 raft_consensus.cc:2781] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:21.442039 21311 raft_consensus.cc:686] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [term 1 LEADER]: Becoming Leader. State: Replica: a9e65d464ca148ddada9f0d08ada07dd, State: Running, Role: LEADER
I20260925 02:53:21.442269 21311 consensus_queue.cc:237] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.443863 21348 tablet_replica.cc:439] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd: TxnStatusTablet state changed. Reason: New leader a9e65d464ca148ddada9f0d08ada07dd. Latest consensus state: current_term: 1 leader_uuid: "a9e65d464ca148ddada9f0d08ada07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } }
I20260925 02:53:21.443982 21348 tablet_replica.cc:442] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:53:21.444360 21350 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:53:21.444469 21350 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:53:21.444720 20779 catalog_manager.cc:5495] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd reported cstate change: term changed from 0 to 1, leader changed from <none> to a9e65d464ca148ddada9f0d08ada07dd (127.12.222.194). New cstate: current_term: 1 leader_uuid: "a9e65d464ca148ddada9f0d08ada07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } health_report { overall_health: UNKNOWN } } }
I20260925 02:53:21.467636 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.463621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:40261 (local address 127.0.0.1:48220)
0925 02:53:21.463774 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:21.463790 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:21.463899 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:21.464272 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:21.464274 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:21.464283 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:53:21.464378 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.464383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.465538 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.465541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.466163 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.466172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.466637 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.466640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.466731 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.466780 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:53:21.467372 (+ 592us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:53:21.467388 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:53:21.467454 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 02:53:21.467532 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 02:53:21.467653 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.463746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48220 (local address 127.12.222.194:40261)
0925 02:53:21.464010 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:21.464018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:21.464036 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:53:21.464084 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:21.464086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:21.464126 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:21.464232 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:53:21.464484 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.465427 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.466296 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.466533 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.466633 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.466925 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:53:21.467274 (+ 349us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:53:21.467392 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:21.467465 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 02:53:21.467510 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":104,"threads_started":1}
I20260925 02:53:21.470284 21229 raft_consensus.cc:1261] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 FOLLOWER]: Refusing update from remote peer a9e65d464ca148ddada9f0d08ada07dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:21.470490 20943 raft_consensus.cc:1261] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65 [term 1 FOLLOWER]: Refusing update from remote peer a9e65d464ca148ddada9f0d08ada07dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:21.470729 21311 consensus_queue.cc:1035] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:21.470970 21348 consensus_queue.cc:1035] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:21.472362 21312 tablet_replica.cc:439] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a: TxnStatusTablet state changed. Reason: New leader a9e65d464ca148ddada9f0d08ada07dd. Latest consensus state: current_term: 1 leader_uuid: "a9e65d464ca148ddada9f0d08ada07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } }
I20260925 02:53:21.472476 21312 tablet_replica.cc:442] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:53:21.472649 21324 tablet_replica.cc:439] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65: TxnStatusTablet state changed. Reason: New leader a9e65d464ca148ddada9f0d08ada07dd. Latest consensus state: current_term: 1 leader_uuid: "a9e65d464ca148ddada9f0d08ada07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } }
I20260925 02:53:21.472770 21324 tablet_replica.cc:442] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:53:21.474224 21348 tablet_replica.cc:439] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9e65d464ca148ddada9f0d08ada07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } }
I20260925 02:53:21.474325 21348 tablet_replica.cc:442] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:53:21.474519 21312 tablet_replica.cc:439] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9e65d464ca148ddada9f0d08ada07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } }
I20260925 02:53:21.474622 21312 tablet_replica.cc:442] T 032c351ad97a404a94e3f7b038bf38c7 P f2d714ec2dab4e23bc6a4bb6e742ef9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:53:21.478044 21311 tablet_replica.cc:439] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9e65d464ca148ddada9f0d08ada07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } }
I20260925 02:53:21.478142 21311 tablet_replica.cc:442] T 032c351ad97a404a94e3f7b038bf38c7 P a9e65d464ca148ddada9f0d08ada07dd: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:53:21.479379 21324 tablet_replica.cc:439] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9e65d464ca148ddada9f0d08ada07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } } }
I20260925 02:53:21.479476 21324 tablet_replica.cc:442] T 032c351ad97a404a94e3f7b038bf38c7 P f970ce150ca54c86a02cf59183955b65: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:53:21.489622 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.486627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:53867 (local address 127.12.222.194:40261)
0925 02:53:21.486793 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:21.486796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:21.486886 (+ 90us) server_negotiation.cc:407] Connection header received
0925 02:53:21.486997 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:21.486999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:21.487031 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:21.487077 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:53:21.487289 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.487690 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.488727 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.488917 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.488985 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.489296 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:53:21.489349 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:53:21.489392 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:21.489507 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 02:53:21.489544 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260925 02:53:21.489696 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.486514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:40261 (local address 127.12.222.193:53867)
0925 02:53:21.486803 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:21.486823 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:21.486913 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:21.487107 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:21.487109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:21.487116 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:53:21.487206 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.487211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.487776 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.487779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.488631 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.488641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.489016 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.489020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.489112 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.489158 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:53:21.489423 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:53:21.489431 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:53:21.489492 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 02:53:21.489572 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":92,"threads_started":1}
I20260925 02:53:21.496708 21229 raft_consensus.cc:1261] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 FOLLOWER]: Refusing update from remote peer f970ce150ca54c86a02cf59183955b65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:21.496888 21084 raft_consensus.cc:1261] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd [term 1 FOLLOWER]: Refusing update from remote peer f970ce150ca54c86a02cf59183955b65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:21.497108 21310 consensus_queue.cc:1035] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:21.497390 21310 consensus_queue.cc:1035] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:21.521353 21345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.516767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:47593 (local address 127.12.222.193:44517)
0925 02:53:21.516921 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:21.516926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:21.517661 (+ 735us) server_negotiation.cc:407] Connection header received
0925 02:53:21.517758 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:21.517762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:21.517821 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:21.517897 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:53:21.518125 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.518885 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.520140 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.520500 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.520594 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.520935 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:53:21.521009 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:53:21.521070 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:21.521184 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 02:53:21.521232 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 02:53:21.521395 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.516189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:44517 (local address 127.12.222.194:47593)
0925 02:53:21.517562 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:21.517575 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:21.517665 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:21.517919 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:21.517922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:21.517928 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:53:21.518017 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.518022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.519036 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.519040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.520015 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.520026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.520616 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.520620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.520709 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.520756 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:53:21.521099 (+ 343us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:53:21.521105 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:53:21.521173 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 02:53:21.521245 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1266,"thread_start_us":116,"threads_started":1}
I20260925 02:53:21.543923 20784 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:21.544078 20784 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:21.545982 20784 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3: Sending DeleteTablet for 3 replicas of tablet a258fd63d7374233805215f8d9eaa04c
I20260925 02:53:21.546509 21064 tablet_service.cc:1514] Processing DeleteTablet for tablet a258fd63d7374233805215f8d9eaa04c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:21 UTC) from {username='oryx', principal='oryx/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:48212
I20260925 02:53:21.546717 20923 tablet_service.cc:1514] Processing DeleteTablet for tablet a258fd63d7374233805215f8d9eaa04c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:21 UTC) from {username='oryx', principal='oryx/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:41722
I20260925 02:53:21.546806 21372 tablet_replica.cc:330] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd: stopping tablet replica
I20260925 02:53:21.546936 21372 raft_consensus.cc:2229] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:21.546990 21209 tablet_service.cc:1514] Processing DeleteTablet for tablet a258fd63d7374233805215f8d9eaa04c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:21 UTC) from {username='oryx', principal='oryx/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:41098
I20260925 02:53:21.547259 21372 raft_consensus.cc:2258] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:21.547443 21373 tablet_replica.cc:330] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65: stopping tablet replica
I20260925 02:53:21.547624 21374 tablet_replica.cc:330] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a: stopping tablet replica
I20260925 02:53:21.547765 21374 raft_consensus.cc:2229] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:21.547931 21373 raft_consensus.cc:2229] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:53:21.548094 21374 raft_consensus.cc:2258] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:21.548146 21372 ts_tablet_manager.cc:1905] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:21.548362 21373 raft_consensus.cc:2258] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:21.550154 21374 ts_tablet_manager.cc:1905] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:21.550645 21373 ts_tablet_manager.cc:1905] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:21.551379 21372 ts_tablet_manager.cc:1918] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:53:21.551467 21372 log.cc:1196] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/a258fd63d7374233805215f8d9eaa04c
I20260925 02:53:21.551750 21372 ts_tablet_manager.cc:1939] T a258fd63d7374233805215f8d9eaa04c P a9e65d464ca148ddada9f0d08ada07dd: Deleting consensus metadata
I20260925 02:53:21.552429 20768 catalog_manager.cc:4841] TS a9e65d464ca148ddada9f0d08ada07dd (127.12.222.194:40261): tablet a258fd63d7374233805215f8d9eaa04c (table test-table [id=c4646ade5a154c49875b560335354111]) successfully deleted
I20260925 02:53:21.554585 21373 ts_tablet_manager.cc:1918] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:53:21.554710 21373 log.cc:1196] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/wal/wals/a258fd63d7374233805215f8d9eaa04c
I20260925 02:53:21.555034 21373 ts_tablet_manager.cc:1939] T a258fd63d7374233805215f8d9eaa04c P f970ce150ca54c86a02cf59183955b65: Deleting consensus metadata
I20260925 02:53:21.555738 20770 catalog_manager.cc:4841] TS f970ce150ca54c86a02cf59183955b65 (127.12.222.193:44517): tablet a258fd63d7374233805215f8d9eaa04c (table test-table [id=c4646ade5a154c49875b560335354111]) successfully deleted
I20260925 02:53:21.556486 21374 ts_tablet_manager.cc:1918] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:53:21.556574 21374 log.cc:1196] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/a258fd63d7374233805215f8d9eaa04c
I20260925 02:53:21.556795 21374 ts_tablet_manager.cc:1939] T a258fd63d7374233805215f8d9eaa04c P f2d714ec2dab4e23bc6a4bb6e742ef9a: Deleting consensus metadata
I20260925 02:53:21.557410 20770 catalog_manager.cc:4841] TS f2d714ec2dab4e23bc6a4bb6e742ef9a (127.12.222.195:44815): tablet a258fd63d7374233805215f8d9eaa04c (table test-table [id=c4646ade5a154c49875b560335354111]) successfully deleted
Sep 25 02:53:21 dist-test-slave-252r krb5kdc[20736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:53:21 dist-test-slave-252r krb5kdc[20736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:21 dist-test-slave-252r krb5kdc[20736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:53:21.574152 21340 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:21.567485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49046 (local address 127.12.222.254:44947)
0925 02:53:21.567624 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:21.567629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:21.567790 (+ 161us) server_negotiation.cc:407] Connection header received
0925 02:53:21.567923 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:21.567925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:21.567968 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:21.568029 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:21.568949 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.569420 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.570547 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.570784 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.570879 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.574042 (+ 3163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49046: BlockingRecv error: recv got EOF from 127.0.0.1:49046 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 02:53:21 dist-test-slave-252r krb5kdc[20736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM
I20260925 02:53:21.583328 21340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.575569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.12.222.254:44947)
0925 02:53:21.575694 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:21.575698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:21.575923 (+ 225us) server_negotiation.cc:407] Connection header received
0925 02:53:21.576065 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:21.576068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:21.576105 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:21.576206 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:21.576830 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.577288 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.578013 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.578227 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.578321 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.580529 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:21.580532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:21.580535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:21.580556 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:21.581839 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:21.582229 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:21.582232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:21.582234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:21.582307 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:21.582570 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:21.582574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:21.582575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:21.582730 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:21.582863 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:21.583079 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 02:53:21.583186 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:53:21.585803 20784 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:53:21.586205 20784 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:53:21.590399 20923 tablet_service.cc:1467] Processing CreateTablet for tablet 655f614aa7b1436c9062ec3fb75fe3a1 (DEFAULT_TABLE table=test-table [id=650fc0bf43ad45fc80410b93b69b80f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:21.590638 20923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 655f614aa7b1436c9062ec3fb75fe3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:21.590670 21209 tablet_service.cc:1467] Processing CreateTablet for tablet 655f614aa7b1436c9062ec3fb75fe3a1 (DEFAULT_TABLE table=test-table [id=650fc0bf43ad45fc80410b93b69b80f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:21.590670 21064 tablet_service.cc:1467] Processing CreateTablet for tablet 655f614aa7b1436c9062ec3fb75fe3a1 (DEFAULT_TABLE table=test-table [id=650fc0bf43ad45fc80410b93b69b80f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:21.590896 21064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 655f614aa7b1436c9062ec3fb75fe3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:21.590898 21209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 655f614aa7b1436c9062ec3fb75fe3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:21.592826 21304 tablet_bootstrap.cc:492] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65: Bootstrap starting.
I20260925 02:53:21.592978 21305 tablet_bootstrap.cc:492] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a: Bootstrap starting.
I20260925 02:53:21.592984 21306 tablet_bootstrap.cc:492] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd: Bootstrap starting.
I20260925 02:53:21.593531 21304 tablet_bootstrap.cc:654] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:21.593808 21305 tablet_bootstrap.cc:654] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:21.593808 21306 tablet_bootstrap.cc:654] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:21.595033 21305 tablet_bootstrap.cc:492] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a: No bootstrap required, opened a new log
I20260925 02:53:21.595136 21305 ts_tablet_manager.cc:1397] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:53:21.595639 21305 raft_consensus.cc:348] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.595724 21305 raft_consensus.cc:374] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:21.595744 21305 raft_consensus.cc:729] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d714ec2dab4e23bc6a4bb6e742ef9a, State: Initialized, Role: FOLLOWER
I20260925 02:53:21.595888 21304 tablet_bootstrap.cc:492] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65: No bootstrap required, opened a new log
I20260925 02:53:21.595865 21305 consensus_queue.cc:260] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.595966 21304 ts_tablet_manager.cc:1397] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 02:53:21.596125 21305 ts_tablet_manager.cc:1428] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:53:21.596376 21304 raft_consensus.cc:348] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.596469 21304 raft_consensus.cc:374] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:21.596496 21304 raft_consensus.cc:729] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f970ce150ca54c86a02cf59183955b65, State: Initialized, Role: FOLLOWER
I20260925 02:53:21.596601 21304 consensus_queue.cc:260] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.596845 21304 ts_tablet_manager.cc:1428] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:53:21.596961 21306 tablet_bootstrap.cc:492] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd: No bootstrap required, opened a new log
I20260925 02:53:21.597029 21306 ts_tablet_manager.cc:1397] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 02:53:21.597456 21306 raft_consensus.cc:348] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.597570 21306 raft_consensus.cc:374] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:21.597601 21306 raft_consensus.cc:729] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e65d464ca148ddada9f0d08ada07dd, State: Initialized, Role: FOLLOWER
I20260925 02:53:21.597711 21306 consensus_queue.cc:260] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.597975 21306 ts_tablet_manager.cc:1428] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:53:21.656365 21310 raft_consensus.cc:482] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:21.656498 21310 raft_consensus.cc:504] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.656868 21310 leader_election.cc:288] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9e65d464ca148ddada9f0d08ada07dd (127.12.222.194:40261), f2d714ec2dab4e23bc6a4bb6e742ef9a (127.12.222.195:44815)
I20260925 02:53:21.657238 21084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "655f614aa7b1436c9062ec3fb75fe3a1" candidate_uuid: "f970ce150ca54c86a02cf59183955b65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e65d464ca148ddada9f0d08ada07dd" is_pre_election: true
I20260925 02:53:21.657275 21229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "655f614aa7b1436c9062ec3fb75fe3a1" candidate_uuid: "f970ce150ca54c86a02cf59183955b65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" is_pre_election: true
I20260925 02:53:21.657388 21084 raft_consensus.cc:2454] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f970ce150ca54c86a02cf59183955b65 in term 0.
I20260925 02:53:21.657413 21229 raft_consensus.cc:2454] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f970ce150ca54c86a02cf59183955b65 in term 0.
I20260925 02:53:21.657680 20875 leader_election.cc:302] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9e65d464ca148ddada9f0d08ada07dd, f970ce150ca54c86a02cf59183955b65; no voters:
I20260925 02:53:21.657860 21310 raft_consensus.cc:2781] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:53:21.657914 21310 raft_consensus.cc:482] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:53:21.657940 21310 raft_consensus.cc:3037] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:21.658828 21310 raft_consensus.cc:504] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.659144 21310 leader_election.cc:288] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [CANDIDATE]: Term 1 election: Requested vote from peers a9e65d464ca148ddada9f0d08ada07dd (127.12.222.194:40261), f2d714ec2dab4e23bc6a4bb6e742ef9a (127.12.222.195:44815)
I20260925 02:53:21.659405 21084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "655f614aa7b1436c9062ec3fb75fe3a1" candidate_uuid: "f970ce150ca54c86a02cf59183955b65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e65d464ca148ddada9f0d08ada07dd"
I20260925 02:53:21.659468 21229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "655f614aa7b1436c9062ec3fb75fe3a1" candidate_uuid: "f970ce150ca54c86a02cf59183955b65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a"
I20260925 02:53:21.659554 21084 raft_consensus.cc:3037] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:21.659583 21229 raft_consensus.cc:3037] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:21.660413 21229 raft_consensus.cc:2454] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f970ce150ca54c86a02cf59183955b65 in term 1.
I20260925 02:53:21.660458 21084 raft_consensus.cc:2454] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f970ce150ca54c86a02cf59183955b65 in term 1.
I20260925 02:53:21.660692 20877 leader_election.cc:302] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f2d714ec2dab4e23bc6a4bb6e742ef9a, f970ce150ca54c86a02cf59183955b65; no voters:
I20260925 02:53:21.660902 21310 raft_consensus.cc:2781] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:21.661044 21310 raft_consensus.cc:686] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [term 1 LEADER]: Becoming Leader. State: Replica: f970ce150ca54c86a02cf59183955b65, State: Running, Role: LEADER
I20260925 02:53:21.661186 21310 consensus_queue.cc:237] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.662704 20779 catalog_manager.cc:5495] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 reported cstate change: term changed from 0 to 1, leader changed from <none> to f970ce150ca54c86a02cf59183955b65 (127.12.222.193). New cstate: current_term: 1 leader_uuid: "f970ce150ca54c86a02cf59183955b65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } health_report { overall_health: UNKNOWN } } }
I20260925 02:53:21.678372 21345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.675586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.12.222.193:44517)
0925 02:53:21.675696 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:21.675701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:21.675736 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:53:21.675877 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:21.675879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:21.675920 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:21.675980 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:53:21.676197 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.676738 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.677511 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.677709 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.677779 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.678003 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:53:21.678062 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:53:21.678112 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:21.678186 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 02:53:21.678224 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:53:21.679913 21229 raft_consensus.cc:1261] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 FOLLOWER]: Refusing update from remote peer f970ce150ca54c86a02cf59183955b65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:21.679958 21084 raft_consensus.cc:1261] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd [term 1 FOLLOWER]: Refusing update from remote peer f970ce150ca54c86a02cf59183955b65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:21.680305 21310 consensus_queue.cc:1035] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:21.680426 21324 consensus_queue.cc:1035] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:21.686512 20784 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:21.686643 20784 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:21.688172 20784 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3: Sending DeleteTablet for 3 replicas of tablet 655f614aa7b1436c9062ec3fb75fe3a1
I20260925 02:53:21.688675 21064 tablet_service.cc:1514] Processing DeleteTablet for tablet 655f614aa7b1436c9062ec3fb75fe3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:21 UTC) from {username='oryx', principal='oryx/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:48212
I20260925 02:53:21.688681 20923 tablet_service.cc:1514] Processing DeleteTablet for tablet 655f614aa7b1436c9062ec3fb75fe3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:21 UTC) from {username='oryx', principal='oryx/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:41722
I20260925 02:53:21.688727 21209 tablet_service.cc:1514] Processing DeleteTablet for tablet 655f614aa7b1436c9062ec3fb75fe3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:21 UTC) from {username='oryx', principal='oryx/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:41098
I20260925 02:53:21.688839 21372 tablet_replica.cc:330] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd: stopping tablet replica
I20260925 02:53:21.688839 21373 tablet_replica.cc:330] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65: stopping tablet replica
I20260925 02:53:21.688954 21373 raft_consensus.cc:2229] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:53:21.688983 21372 raft_consensus.cc:2229] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:21.689111 21372 raft_consensus.cc:2258] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:21.689132 21373 raft_consensus.cc:2258] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:21.689286 21374 tablet_replica.cc:330] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a: stopping tablet replica
I20260925 02:53:21.689409 21374 raft_consensus.cc:2229] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:21.689541 21374 raft_consensus.cc:2258] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:21.690047 21373 ts_tablet_manager.cc:1905] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:21.690464 21374 ts_tablet_manager.cc:1905] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:21.690719 20784 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:53:21.691473 20784 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:53:21.692636 21373 ts_tablet_manager.cc:1918] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:21.692711 21373 log.cc:1196] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/wal/wals/655f614aa7b1436c9062ec3fb75fe3a1
I20260925 02:53:21.692710 21372 ts_tablet_manager.cc:1905] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:21.692847 21374 ts_tablet_manager.cc:1918] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:21.692994 21374 log.cc:1196] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/655f614aa7b1436c9062ec3fb75fe3a1
I20260925 02:53:21.693009 21373 ts_tablet_manager.cc:1939] T 655f614aa7b1436c9062ec3fb75fe3a1 P f970ce150ca54c86a02cf59183955b65: Deleting consensus metadata
I20260925 02:53:21.693250 21374 ts_tablet_manager.cc:1939] T 655f614aa7b1436c9062ec3fb75fe3a1 P f2d714ec2dab4e23bc6a4bb6e742ef9a: Deleting consensus metadata
I20260925 02:53:21.693974 20770 catalog_manager.cc:4841] TS f2d714ec2dab4e23bc6a4bb6e742ef9a (127.12.222.195:44815): tablet 655f614aa7b1436c9062ec3fb75fe3a1 (table test-table [id=650fc0bf43ad45fc80410b93b69b80f0]) successfully deleted
I20260925 02:53:21.694118 20770 catalog_manager.cc:4841] TS f970ce150ca54c86a02cf59183955b65 (127.12.222.193:44517): tablet 655f614aa7b1436c9062ec3fb75fe3a1 (table test-table [id=650fc0bf43ad45fc80410b93b69b80f0]) successfully deleted
I20260925 02:53:21.695103 21372 ts_tablet_manager.cc:1918] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:21.695206 21372 log.cc:1196] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/655f614aa7b1436c9062ec3fb75fe3a1
I20260925 02:53:21.695472 21372 ts_tablet_manager.cc:1939] T 655f614aa7b1436c9062ec3fb75fe3a1 P a9e65d464ca148ddada9f0d08ada07dd: Deleting consensus metadata
I20260925 02:53:21.696091 20768 catalog_manager.cc:4841] TS a9e65d464ca148ddada9f0d08ada07dd (127.12.222.194:40261): tablet 655f614aa7b1436c9062ec3fb75fe3a1 (table test-table [id=650fc0bf43ad45fc80410b93b69b80f0]) successfully deleted
I20260925 02:53:21.700950 20923 tablet_service.cc:1467] Processing CreateTablet for tablet 6d08818746584762a9a80c323f072f8d (DEFAULT_TABLE table=test-table [id=701a5e8d82c943b8b0fe8518a0470d4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:21.701192 20923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d08818746584762a9a80c323f072f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:21.701200 21209 tablet_service.cc:1467] Processing CreateTablet for tablet 6d08818746584762a9a80c323f072f8d (DEFAULT_TABLE table=test-table [id=701a5e8d82c943b8b0fe8518a0470d4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:21.701408 21209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d08818746584762a9a80c323f072f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:21.702997 21304 tablet_bootstrap.cc:492] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65: Bootstrap starting.
I20260925 02:53:21.703290 21305 tablet_bootstrap.cc:492] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a: Bootstrap starting.
I20260925 02:53:21.703426 21064 tablet_service.cc:1467] Processing CreateTablet for tablet 6d08818746584762a9a80c323f072f8d (DEFAULT_TABLE table=test-table [id=701a5e8d82c943b8b0fe8518a0470d4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:21.703637 21064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d08818746584762a9a80c323f072f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:21.703702 21304 tablet_bootstrap.cc:654] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:21.704044 21305 tablet_bootstrap.cc:654] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:21.705175 21305 tablet_bootstrap.cc:492] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a: No bootstrap required, opened a new log
I20260925 02:53:21.705260 21305 ts_tablet_manager.cc:1397] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:53:21.705639 21306 tablet_bootstrap.cc:492] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd: Bootstrap starting.
I20260925 02:53:21.705664 21305 raft_consensus.cc:348] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.705755 21305 raft_consensus.cc:374] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:21.705780 21305 raft_consensus.cc:729] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d714ec2dab4e23bc6a4bb6e742ef9a, State: Initialized, Role: FOLLOWER
I20260925 02:53:21.705884 21305 consensus_queue.cc:260] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.706082 21305 ts_tablet_manager.cc:1428] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:53:21.706465 21306 tablet_bootstrap.cc:654] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:21.707444 21306 tablet_bootstrap.cc:492] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd: No bootstrap required, opened a new log
I20260925 02:53:21.707544 21306 ts_tablet_manager.cc:1397] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:53:21.707867 21306 raft_consensus.cc:348] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.707973 21306 raft_consensus.cc:374] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:21.708009 21306 raft_consensus.cc:729] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e65d464ca148ddada9f0d08ada07dd, State: Initialized, Role: FOLLOWER
I20260925 02:53:21.708130 21306 consensus_queue.cc:260] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.708417 21306 ts_tablet_manager.cc:1428] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:53:21.713907 21304 tablet_bootstrap.cc:492] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65: No bootstrap required, opened a new log
I20260925 02:53:21.713979 21304 ts_tablet_manager.cc:1397] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260925 02:53:21.714341 21304 raft_consensus.cc:348] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.714463 21304 raft_consensus.cc:374] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:21.714522 21304 raft_consensus.cc:729] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f970ce150ca54c86a02cf59183955b65, State: Initialized, Role: FOLLOWER
I20260925 02:53:21.714676 21304 consensus_queue.cc:260] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.714936 21304 ts_tablet_manager.cc:1428] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:53:21.726145 21312 raft_consensus.cc:482] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:21.726227 21312 raft_consensus.cc:504] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.727093 21312 leader_election.cc:288] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f970ce150ca54c86a02cf59183955b65 (127.12.222.193:44517), a9e65d464ca148ddada9f0d08ada07dd (127.12.222.194:40261)
I20260925 02:53:21.730837 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.727272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:40261 (local address 127.12.222.195:44843)
0925 02:53:21.727534 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:21.727546 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:21.727624 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:21.727836 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:21.727839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:21.727856 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:21.727999 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.728005 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.728811 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.728815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.729744 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.729752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.730499 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.730502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.730596 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.730621 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:53:21.730676 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 02:53:21.730720 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":38,"thread_start_us":50,"threads_started":1}
I20260925 02:53:21.730844 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.727360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:44843 (local address 127.12.222.194:40261)
0925 02:53:21.727478 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:21.727482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:21.727611 (+ 129us) server_negotiation.cc:407] Connection header received
0925 02:53:21.727695 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:21.727698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:21.727741 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:21.727809 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:21.728117 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.728676 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.729907 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.730405 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.730488 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.730523 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:21.730706 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 02:53:21.730757 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:53:21.731238 21084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d08818746584762a9a80c323f072f8d" candidate_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e65d464ca148ddada9f0d08ada07dd" is_pre_election: true
I20260925 02:53:21.731350 21084 raft_consensus.cc:2454] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2d714ec2dab4e23bc6a4bb6e742ef9a in term 0.
I20260925 02:53:21.731405 21398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.727272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:44517 (local address 127.12.222.195:47411)
0925 02:53:21.727620 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:21.727634 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:21.727729 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:21.728002 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:21.728004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:21.728018 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:21.728130 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.728138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.728879 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.728882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.729723 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.729733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.731001 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.731006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.731084 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.731099 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 02:53:21.731215 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 02:53:21.731280 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":114,"thread_start_us":85,"threads_started":1}
I20260925 02:53:21.731434 21345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.727388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:47411 (local address 127.12.222.193:44517)
0925 02:53:21.727478 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:21.727482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:21.727723 (+ 241us) server_negotiation.cc:407] Connection header received
0925 02:53:21.727839 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:21.727842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:21.727878 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:21.727966 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:21.728270 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.728772 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.729834 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.730358 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.730491 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.730533 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:21.731260 (+ 727us) server_negotiation.cc:299] Negotiation successful
0925 02:53:21.731327 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:53:21.731766 20943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d08818746584762a9a80c323f072f8d" candidate_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f970ce150ca54c86a02cf59183955b65" is_pre_election: true
I20260925 02:53:21.731866 20943 raft_consensus.cc:2454] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2d714ec2dab4e23bc6a4bb6e742ef9a in term 0.
I20260925 02:53:21.731873 21160 leader_election.cc:302] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9e65d464ca148ddada9f0d08ada07dd, f2d714ec2dab4e23bc6a4bb6e742ef9a; no voters:
I20260925 02:53:21.732105 21312 raft_consensus.cc:2781] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:53:21.732165 21312 raft_consensus.cc:482] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:53:21.732213 21312 raft_consensus.cc:3037] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:21.733008 21312 raft_consensus.cc:504] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.733302 21312 leader_election.cc:288] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [CANDIDATE]: Term 1 election: Requested vote from peers f970ce150ca54c86a02cf59183955b65 (127.12.222.193:44517), a9e65d464ca148ddada9f0d08ada07dd (127.12.222.194:40261)
I20260925 02:53:21.733614 21084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d08818746584762a9a80c323f072f8d" candidate_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e65d464ca148ddada9f0d08ada07dd"
I20260925 02:53:21.733615 20943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d08818746584762a9a80c323f072f8d" candidate_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f970ce150ca54c86a02cf59183955b65"
I20260925 02:53:21.733718 20943 raft_consensus.cc:3037] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:21.733718 21084 raft_consensus.cc:3037] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:21.734622 20943 raft_consensus.cc:2454] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2d714ec2dab4e23bc6a4bb6e742ef9a in term 1.
I20260925 02:53:21.734622 21084 raft_consensus.cc:2454] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2d714ec2dab4e23bc6a4bb6e742ef9a in term 1.
I20260925 02:53:21.734869 21162 leader_election.cc:302] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f2d714ec2dab4e23bc6a4bb6e742ef9a, f970ce150ca54c86a02cf59183955b65; no voters:
I20260925 02:53:21.735024 21312 raft_consensus.cc:2781] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:21.735229 21312 raft_consensus.cc:686] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 LEADER]: Becoming Leader. State: Replica: f2d714ec2dab4e23bc6a4bb6e742ef9a, State: Running, Role: LEADER
I20260925 02:53:21.735453 21312 consensus_queue.cc:237] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } }
I20260925 02:53:21.737327 20779 catalog_manager.cc:5495] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a reported cstate change: term changed from 0 to 1, leader changed from <none> to f2d714ec2dab4e23bc6a4bb6e742ef9a (127.12.222.195). New cstate: current_term: 1 leader_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2d714ec2dab4e23bc6a4bb6e742ef9a" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 44815 } health_report { overall_health: HEALTHY } } }
I20260925 02:53:21.759547 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.756406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.12.222.195:44815)
0925 02:53:21.756517 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:21.756524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:21.756538 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:53:21.756705 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:21.756708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:21.756759 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:21.756857 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:53:21.757072 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.757512 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.758255 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.758457 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.758576 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.758821 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:53:21.759211 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:53:21.759331 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:21.759387 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 02:53:21.759439 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:53:21.768282 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.765337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:33347 (local address 127.12.222.194:40261)
0925 02:53:21.765457 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:21.765461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:21.765629 (+ 168us) server_negotiation.cc:407] Connection header received
0925 02:53:21.765703 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:21.765705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:21.765741 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:21.765799 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:53:21.766043 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.766500 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.767299 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.767501 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.767597 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.767852 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:53:21.767913 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:53:21.767988 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:21.768102 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 02:53:21.768145 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:53:21.768319 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.765277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:40261 (local address 127.12.222.195:33347)
0925 02:53:21.765554 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:21.765566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:21.765644 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:21.765819 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:21.765821 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:21.765826 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:53:21.765924 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.765932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.766597 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.766602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.767189 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.767198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.767608 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.767611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.767693 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.767728 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:53:21.768006 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:53:21.768012 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:53:21.768088 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 02:53:21.768153 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":108,"threads_started":1}
I20260925 02:53:21.773279 21084 raft_consensus.cc:1261] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd [term 1 FOLLOWER]: Refusing update from remote peer f2d714ec2dab4e23bc6a4bb6e742ef9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:21.773274 20943 raft_consensus.cc:1261] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65 [term 1 FOLLOWER]: Refusing update from remote peer f2d714ec2dab4e23bc6a4bb6e742ef9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:21.773715 21312 consensus_queue.cc:1035] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f970ce150ca54c86a02cf59183955b65" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:21.774223 21399 consensus_queue.cc:1035] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e65d464ca148ddada9f0d08ada07dd" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:21.788851 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.786088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:40521 (local address 127.12.222.195:44815)
0925 02:53:21.786197 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:21.786201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:21.786213 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:53:21.786275 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:21.786278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:21.786335 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:21.786418 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:53:21.786635 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.787083 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.787808 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.788108 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.788221 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.788460 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:53:21.788526 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:53:21.788590 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:21.788689 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 02:53:21.788724 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:53:21.788868 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:21.785693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:44815 (local address 127.12.222.194:40521)
0925 02:53:21.785817 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:21.785831 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:21.785919 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:21.786454 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:21.786458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:21.786468 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:53:21.786549 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.786553 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.787171 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.787173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.787697 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:21.787704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:21.788250 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:21.788256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:21.788334 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:21.788355 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:53:21.788610 (+ 255us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:53:21.788617 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:53:21.788676 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 02:53:21.788754 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 02:53:21.806604 20784 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:21.806749 20784 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:21.808652 20784 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0424e79fe3ce45de984a88a5339e5da3: Sending DeleteTablet for 3 replicas of tablet 6d08818746584762a9a80c323f072f8d
I20260925 02:53:21.809127 20923 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d08818746584762a9a80c323f072f8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:21 UTC) from {username='oryx', principal='oryx/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:41722
I20260925 02:53:21.809158 21064 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d08818746584762a9a80c323f072f8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:21 UTC) from {username='oryx', principal='oryx/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:48212
I20260925 02:53:21.809309 21372 tablet_replica.cc:330] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd: stopping tablet replica
I20260925 02:53:21.809309 21373 tablet_replica.cc:330] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65: stopping tablet replica
I20260925 02:53:21.809409 21373 raft_consensus.cc:2229] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:21.809420 21372 raft_consensus.cc:2229] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:21.809527 21372 raft_consensus.cc:2258] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:21.809528 21373 raft_consensus.cc:2258] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:21.810082 21209 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d08818746584762a9a80c323f072f8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:21 UTC) from {username='oryx', principal='oryx/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:41098
I20260925 02:53:21.810155 21372 ts_tablet_manager.cc:1905] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:21.810245 21374 tablet_replica.cc:330] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a: stopping tablet replica
I20260925 02:53:21.810344 21374 raft_consensus.cc:2229] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:53:21.810588 21374 raft_consensus.cc:2258] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:21.811201 21373 ts_tablet_manager.cc:1905] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:21.811199 21374 ts_tablet_manager.cc:1905] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:21.812713 21372 ts_tablet_manager.cc:1918] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:53:21.812779 21372 log.cc:1196] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/6d08818746584762a9a80c323f072f8d
I20260925 02:53:21.813058 21372 ts_tablet_manager.cc:1939] T 6d08818746584762a9a80c323f072f8d P a9e65d464ca148ddada9f0d08ada07dd: Deleting consensus metadata
I20260925 02:53:21.813441 21374 ts_tablet_manager.cc:1918] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:53:21.813438 21373 ts_tablet_manager.cc:1918] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:53:21.813503 21373 log.cc:1196] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-0/wal/wals/6d08818746584762a9a80c323f072f8d
I20260925 02:53:21.813609 20768 catalog_manager.cc:4841] TS a9e65d464ca148ddada9f0d08ada07dd (127.12.222.194:40261): tablet 6d08818746584762a9a80c323f072f8d (table test-table [id=701a5e8d82c943b8b0fe8518a0470d4c]) successfully deleted
I20260925 02:53:21.813740 21373 ts_tablet_manager.cc:1939] T 6d08818746584762a9a80c323f072f8d P f970ce150ca54c86a02cf59183955b65: Deleting consensus metadata
I20260925 02:53:21.813798 21374 log.cc:1196] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/6d08818746584762a9a80c323f072f8d
I20260925 02:53:21.814028 21374 ts_tablet_manager.cc:1939] T 6d08818746584762a9a80c323f072f8d P f2d714ec2dab4e23bc6a4bb6e742ef9a: Deleting consensus metadata
I20260925 02:53:21.814430 20770 catalog_manager.cc:4841] TS f970ce150ca54c86a02cf59183955b65 (127.12.222.193:44517): tablet 6d08818746584762a9a80c323f072f8d (table test-table [id=701a5e8d82c943b8b0fe8518a0470d4c]) successfully deleted
I20260925 02:53:21.814571 20770 catalog_manager.cc:4841] TS f2d714ec2dab4e23bc6a4bb6e742ef9a (127.12.222.195:44815): tablet 6d08818746584762a9a80c323f072f8d (table test-table [id=701a5e8d82c943b8b0fe8518a0470d4c]) successfully deleted
I20260925 02:53:21.815203 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 20857
I20260925 02:53:21.822736 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 20999
I20260925 02:53:21.831081 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 21142
I20260925 02:53:21.836923 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 20749
2026-09-25T02:53:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3053 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:53:21 dist-test-slave-252r krb5kdc[21409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:53:21 dist-test-slave-252r krb5kdc[21409](info): set up 2 sockets
Sep 25 02:53:21 dist-test-slave-252r krb5kdc[21409](info): commencing operation
krb5kdc: starting...
W20260925 02:53:22.796546 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:53:22 dist-test-slave-252r krb5kdc[21409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:53:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:53:22.901052 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:34089
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:34089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:23.001935 21423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:23.002148 21423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:23.002198 21423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:23.006157 21423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:23.006204 21423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:23.006222 21423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:23.006291 21423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:23.006335 21423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:23.010783 21423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:34089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:34089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:23.011641 21423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:23.012581 21423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:23.020335 21431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:23.020330 21428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:23.020553 21423 server_base.cc:1034] running on GCE node
W20260925 02:53:23.020637 21429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:23.021154 21423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:23.022055 21423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:23.023211 21423 hybrid_clock.cc:648] HybridClock initialized: now 1790304803023186 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T02:53:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1200 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:53:23 dist-test-slave-252r krb5kdc[21436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:53:23 dist-test-slave-252r krb5kdc[21436](info): set up 2 sockets
Sep 25 02:53:23 dist-test-slave-252r krb5kdc[21436](info): commencing operation
krb5kdc: starting...
W20260925 02:53:24.000618 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:53:24 dist-test-slave-252r krb5kdc[21436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:53:24.106880 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:43351
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:36179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:43351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:24.206825 21450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:24.207062 21450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:24.207125 21450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:24.210719 21450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:24.210784 21450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:24.210817 21450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:24.210867 21450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:24.210901 21450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:24.215539 21450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:36179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:43351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:43351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:24.216429 21450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:24.217307 21450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:24.224997 21458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:24.225008 21455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:24.225008 21456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:24.225411 21450 server_base.cc:1034] running on GCE node
I20260925 02:53:24.225875 21450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:24.226830 21450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:24.227990 21450 hybrid_clock.cc:648] HybridClock initialized: now 1790304804227973 us; error 28 us; skew 500 ppm
Sep 25 02:53:24 dist-test-slave-252r krb5kdc[21436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304804, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:24.230540 21450 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:53:24.231281 21450 webserver.cc:460] Webserver started at http://127.12.222.254:42325/ using document root <none> and password file <none>
I20260925 02:53:24.231719 21450 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:24.231768 21450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:24.231907 21450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:24.233654 21450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "bc15923d224e4559b1f509272e0e6c42"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-252r"
server_key: "89d684cbc9caa8758c492de7881683a2"
server_key_iv: "b9192efa5f9bd8ba56f0b8a3180afc26"
server_key_version: "encryptionkey@0"
I20260925 02:53:24.234098 21450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "bc15923d224e4559b1f509272e0e6c42"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-252r"
server_key: "89d684cbc9caa8758c492de7881683a2"
server_key_iv: "b9192efa5f9bd8ba56f0b8a3180afc26"
server_key_version: "encryptionkey@0"
I20260925 02:53:24.237306 21450 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260925 02:53:24.239274 21465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:24.240417 21450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:53:24.240532 21450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "bc15923d224e4559b1f509272e0e6c42"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-252r"
server_key: "89d684cbc9caa8758c492de7881683a2"
server_key_iv: "b9192efa5f9bd8ba56f0b8a3180afc26"
server_key_version: "encryptionkey@0"
I20260925 02:53:24.240623 21450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:24.261660 21450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:24.264113 21450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:24.264314 21450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:24.270540 21450 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:43351
I20260925 02:53:24.270555 21520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:43351 every 8 connection(s)
I20260925 02:53:24.271449 21450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:24.273766 21521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:24.278558 21521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42: Bootstrap starting.
I20260925 02:53:24.280646 21521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:24.281294 21521 log.cc:824] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:24.281612 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 21450
I20260925 02:53:24.281745 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:53:24.282008 13179 external_mini_cluster.cc:1428] Setting key a3fcaee1e3e0825fa66307cda23ca988
I20260925 02:53:24.283154 21521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42: No bootstrap required, opened a new log
I20260925 02:53:24.287286 21521 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc15923d224e4559b1f509272e0e6c42" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43351 } }
I20260925 02:53:24.287469 21521 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:24.287520 21521 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc15923d224e4559b1f509272e0e6c42, State: Initialized, Role: FOLLOWER
I20260925 02:53:24.287890 21521 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc15923d224e4559b1f509272e0e6c42" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43351 } }
I20260925 02:53:24.288028 21521 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:24.288081 21521 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:24.288160 21521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:24.289060 21521 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc15923d224e4559b1f509272e0e6c42" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43351 } }
I20260925 02:53:24.289361 21521 leader_election.cc:302] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc15923d224e4559b1f509272e0e6c42; no voters:
I20260925 02:53:24.289615 21521 leader_election.cc:288] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:53:24.289739 21526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:24.289997 21526 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [term 1 LEADER]: Becoming Leader. State: Replica: bc15923d224e4559b1f509272e0e6c42, State: Running, Role: LEADER
I20260925 02:53:24.290297 21526 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc15923d224e4559b1f509272e0e6c42" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43351 } }
Sep 25 02:53:24 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:24.290719 21521 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:24.291781 21527 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc15923d224e4559b1f509272e0e6c42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc15923d224e4559b1f509272e0e6c42" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43351 } } }
I20260925 02:53:24.291791 21528 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc15923d224e4559b1f509272e0e6c42. Latest consensus state: current_term: 1 leader_uuid: "bc15923d224e4559b1f509272e0e6c42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc15923d224e4559b1f509272e0e6c42" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43351 } } }
I20260925 02:53:24.291903 21527 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:24.291934 21528 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:24.292351 21536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:24.296512 21536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:24.296649 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:24.282936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.12.222.254:43351)
0925 02:53:24.283352 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:24.283369 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:24.283404 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:53:24.284217 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:24.284261 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:24.284611 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:24.284932 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:24.285511 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:24.286960 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:24.287883 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:24.288566 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:24.288719 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:24.291350 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:24.291357 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:24.291373 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:24.291406 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:24.293666 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:24.294051 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:24.294056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:24.294060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:24.294191 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:24.294512 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:24.294516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:24.294517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:24.294992 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:24.295177 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:24.295495 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 02:53:24.295762 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
I20260925 02:53:24.301705 21536 catalog_manager.cc:1347] Generated new cluster ID: b3d6e5dd36d34498bf863d0cfdca9b0e
I20260925 02:53:24.301774 21536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:24.307809 21536 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:24.308643 21536 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:24.321765 21536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42: Generated new TSK 0
I20260925 02:53:24.322315 21536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
I20260925 02:53:24.352775 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:43351
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:36179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:24.454455 21548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:24.454653 21548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:24.454715 21548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:24.458618 21548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:24.458662 21548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:24.458710 21548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:53:24.463281 21548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:36179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:24.464146 21548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:24.465018 21548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:24.473397 21556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:24.473454 21548 server_base.cc:1034] running on GCE node
W20260925 02:53:24.473554 21553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:24.473465 21554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:24.473922 21548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:24.474447 21548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:24.475642 21548 hybrid_clock.cc:648] HybridClock initialized: now 1790304804475626 us; error 37 us; skew 500 ppm
Sep 25 02:53:24 dist-test-slave-252r krb5kdc[21436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304804, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:24.478248 21548 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:53:24.478955 21548 webserver.cc:460] Webserver started at http://127.12.222.193:35739/ using document root <none> and password file <none>
I20260925 02:53:24.479424 21548 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:24.479507 21548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:24.479710 21548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:24.481491 21548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "46ab7dd4116a4780959f9fc0b59d9e8c"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-252r"
server_key: "5edd8751c8643b1dd41f6c51ad09c1ae"
server_key_iv: "bcdee6d067a9987d19689299e28d5c25"
server_key_version: "encryptionkey@0"
I20260925 02:53:24.482041 21548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "46ab7dd4116a4780959f9fc0b59d9e8c"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-252r"
server_key: "5edd8751c8643b1dd41f6c51ad09c1ae"
server_key_iv: "bcdee6d067a9987d19689299e28d5c25"
server_key_version: "encryptionkey@0"
I20260925 02:53:24.485219 21548 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 02:53:24.487275 21564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:24.488346 21548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:53:24.488456 21548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "46ab7dd4116a4780959f9fc0b59d9e8c"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-252r"
server_key: "5edd8751c8643b1dd41f6c51ad09c1ae"
server_key_iv: "bcdee6d067a9987d19689299e28d5c25"
server_key_version: "encryptionkey@0"
I20260925 02:53:24.488546 21548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:24.502357 21548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:24.505139 21548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:24.505306 21548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:24.505816 21548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:24.506523 21548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:24.506614 21548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:24.506672 21548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:24.506716 21548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:24.514147 21548 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:43085
I20260925 02:53:24.514205 21682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:43085 every 8 connection(s)
I20260925 02:53:24.514967 21548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:53:24.517755 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 21548
I20260925 02:53:24.517890 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:53:24.518222 13179 external_mini_cluster.cc:1428] Setting key 74f7ad7be24e1137fe35467b8723eb84
Sep 25 02:53:24 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304804, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.222.194@KRBTEST.COM; defaulting to no policy
I20260925 02:53:24.529173 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:24.516728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:46533 (local address 127.12.222.254:43351)
0925 02:53:24.516880 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:24.516884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:24.517859 (+ 975us) server_negotiation.cc:407] Connection header received
0925 02:53:24.518873 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:24.518877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:24.518942 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:24.519008 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:24.520709 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:24.521287 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:24.522484 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:24.522876 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:24.523001 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:24.525635 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:24.525639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:24.525641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:24.525664 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:24.527129 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:24.527632 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:24.527635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:24.527637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:24.527744 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:24.528075 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:24.528079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:24.528081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:24.528299 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:24.528408 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:24.528885 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 02:53:24.529007 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:53:24.529925 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:24.517074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:46533)
0925 02:53:24.517687 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:24.517743 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:24.518679 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:24.519491 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:24.519496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:24.519928 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:24.520571 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:24.520590 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:24.521416 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:24.521419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:24.522323 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:24.522334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:24.523025 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:24.523028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:24.523159 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:24.523667 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:24.523681 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:24.525466 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:24.527251 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:24.527256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:24.527268 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:24.527513 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:24.527832 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:24.527835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:24.527838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:24.527978 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:24.528408 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:24.528414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:24.528634 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 02:53:24.528843 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 02:53:24.529135 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.194@KRBTEST.COM" created.
I20260925 02:53:24.531061 21683 heartbeater.cc:347] Connected to a master server at 127.12.222.254:43351
I20260925 02:53:24.531292 21683 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:24.531786 21683 heartbeater.cc:511] Master 127.12.222.254:43351 requested a full tablet report, sending...
I20260925 02:53:24.533339 21483 ts_manager.cc:194] Registered new tserver with Master: 46ab7dd4116a4780959f9fc0b59d9e8c (127.12.222.193:43085)
I20260925 02:53:24.534400 21483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:46533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
I20260925 02:53:24.547477 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:43351
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:36179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:24.647547 21688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:24.647740 21688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:24.647809 21688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:24.651358 21688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:24.651436 21688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:24.651520 21688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:53:24.656149 21688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:36179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:24.657006 21688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:24.657855 21688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:24.667174 21693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:24.667224 21694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:24.667654 21696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:24.668010 21688 server_base.cc:1034] running on GCE node
I20260925 02:53:24.668460 21688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:24.669162 21688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:24.670318 21688 hybrid_clock.cc:648] HybridClock initialized: now 1790304804670264 us; error 65 us; skew 500 ppm
Sep 25 02:53:24 dist-test-slave-252r krb5kdc[21436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304804, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:24.674095 21688 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:53:24.675014 21688 webserver.cc:460] Webserver started at http://127.12.222.194:33407/ using document root <none> and password file <none>
I20260925 02:53:24.675835 21688 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:24.675948 21688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:24.676218 21688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:24.679203 21688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "0f65e960d86445fcae3b3ecf1e0113d4"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-252r"
server_key: "6b28d0e189416acd71f50a94c66f45a9"
server_key_iv: "5c84cbd34d571f4ddbc157f107b4a31f"
server_key_version: "encryptionkey@0"
I20260925 02:53:24.679988 21688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "0f65e960d86445fcae3b3ecf1e0113d4"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-252r"
server_key: "6b28d0e189416acd71f50a94c66f45a9"
server_key_iv: "5c84cbd34d571f4ddbc157f107b4a31f"
server_key_version: "encryptionkey@0"
I20260925 02:53:24.684437 21688 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:53:24.687382 21703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:24.688746 21688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:53:24.688871 21688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "0f65e960d86445fcae3b3ecf1e0113d4"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-252r"
server_key: "6b28d0e189416acd71f50a94c66f45a9"
server_key_iv: "5c84cbd34d571f4ddbc157f107b4a31f"
server_key_version: "encryptionkey@0"
I20260925 02:53:24.688987 21688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:24.715219 21688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:24.718153 21688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:24.718325 21688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:24.718808 21688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:24.719532 21688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:24.719597 21688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:24.719658 21688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:24.719703 21688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:24.727809 21688 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:43881
I20260925 02:53:24.727833 21821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:43881 every 8 connection(s)
I20260925 02:53:24.728646 21688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:53:24.732652 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 21688
I20260925 02:53:24.732779 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:53:24.733083 13179 external_mini_cluster.cc:1428] Setting key 4102facba36b40e75bdf20beec456f83
Sep 25 02:53:24 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304804, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:24.741580 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:24.730359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:52077 (local address 127.12.222.254:43351)
0925 02:53:24.730549 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:24.730554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:24.731504 (+ 950us) server_negotiation.cc:407] Connection header received
0925 02:53:24.732357 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:24.732360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:24.732403 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:24.732480 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:24.733755 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:24.734310 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:24.735385 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:24.735615 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:24.735723 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:24.738278 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:24.738286 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:24.738289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:24.738310 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:24.739637 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:24.740121 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:24.740124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:24.740126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:24.740201 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:24.740551 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:24.740555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:24.740556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:24.740717 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:24.740827 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:24.741292 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 02:53:24.741408 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
WARNING: no policy specified for kudu/127.12.222.195@KRBTEST.COM; defaulting to no policy
I20260925 02:53:24.742321 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:24.730774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:52077)
0925 02:53:24.731358 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:24.731396 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:24.732098 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:24.732680 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:24.732689 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:24.733121 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:24.733633 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:24.733646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:24.734426 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:24.734429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:24.735255 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:24.735264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:24.735760 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:24.735764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:24.735925 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:24.736587 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:24.736610 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:24.738117 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:24.739764 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:24.739769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:24.739781 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:24.740021 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:24.740323 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:24.740326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:24.740327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:24.740457 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:24.740828 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:24.740835 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:24.741070 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 02:53:24.741247 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 02:53:24.741532 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":107,"threads_started":1}
I20260925 02:53:24.743397 21822 heartbeater.cc:347] Connected to a master server at 127.12.222.254:43351
I20260925 02:53:24.743582 21822 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:24.744045 21822 heartbeater.cc:511] Master 127.12.222.254:43351 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.195@KRBTEST.COM" created.
I20260925 02:53:24.745154 21483 ts_manager.cc:194] Registered new tserver with Master: 0f65e960d86445fcae3b3ecf1e0113d4 (127.12.222.194:43881)
I20260925 02:53:24.745699 21483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:52077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
I20260925 02:53:24.759338 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:43351
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:36179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:24.859772 21827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:24.859974 21827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:24.860057 21827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:24.863773 21827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:24.863814 21827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:24.863901 21827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:53:24.868413 21827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:36179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:24.869215 21827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:24.870064 21827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:24.878261 21834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:24.878279 21833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:24.878297 21836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:24.878343 21827 server_base.cc:1034] running on GCE node
I20260925 02:53:24.878834 21827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:24.879351 21827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:24.880522 21827 hybrid_clock.cc:648] HybridClock initialized: now 1790304804880504 us; error 27 us; skew 500 ppm
Sep 25 02:53:24 dist-test-slave-252r krb5kdc[21436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304804, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:24.882987 21827 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:53:24.883621 21827 webserver.cc:460] Webserver started at http://127.12.222.195:40517/ using document root <none> and password file <none>
I20260925 02:53:24.884073 21827 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:24.884125 21827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:24.884317 21827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:24.885958 21827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "e1f2ca51eb1c4b73b11ea3ef5241c769"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-252r"
server_key: "c6ca1b4966f30335f3b265ce6be396c3"
server_key_iv: "5d5c3178152b990abce2b634fc261fb8"
server_key_version: "encryptionkey@0"
I20260925 02:53:24.886443 21827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "e1f2ca51eb1c4b73b11ea3ef5241c769"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-252r"
server_key: "c6ca1b4966f30335f3b265ce6be396c3"
server_key_iv: "5d5c3178152b990abce2b634fc261fb8"
server_key_version: "encryptionkey@0"
I20260925 02:53:24.889446 21827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:53:24.891588 21843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:24.892808 21827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:53:24.892908 21827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "e1f2ca51eb1c4b73b11ea3ef5241c769"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-252r"
server_key: "c6ca1b4966f30335f3b265ce6be396c3"
server_key_iv: "5d5c3178152b990abce2b634fc261fb8"
server_key_version: "encryptionkey@0"
I20260925 02:53:24.893008 21827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:24.911156 21827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:24.913435 21827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:24.913599 21827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:24.914142 21827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:24.914809 21827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:24.914851 21827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:24.914908 21827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:24.914949 21827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:24.922652 21827 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:45483
I20260925 02:53:24.922685 21957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:45483 every 8 connection(s)
I20260925 02:53:24.923353 21827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:53:24.924429 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 21827
I20260925 02:53:24.924556 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:53:24.924856 13179 external_mini_cluster.cc:1428] Setting key ece031634cd9291fd9984fe441c9bce9
Sep 25 02:53:24 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304804, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:24.934780 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:24.924974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:47929 (local address 127.12.222.254:43351)
0925 02:53:24.925104 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:24.925111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:24.926178 (+ 1067us) server_negotiation.cc:407] Connection header received
0925 02:53:24.927114 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:24.927117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:24.927158 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:24.927220 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:24.928151 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:24.928714 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:24.929710 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:24.929913 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:24.929990 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:24.931935 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:24.931938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:24.931940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:24.931960 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:24.933282 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:24.933676 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:24.933678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:24.933679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:24.933743 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:24.934019 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:24.934021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:24.934022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:24.934160 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:24.934224 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:24.934594 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 02:53:24.934675 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:53:24.935413 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:24.925329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.195:47929)
0925 02:53:24.926031 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:24.926077 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:24.926929 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:24.927357 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:24.927362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:24.927655 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:24.928036 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:24.928048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:24.928811 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:24.928813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:24.929609 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:24.929618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:24.930042 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:24.930045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:24.930157 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:24.930559 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:24.930573 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:24.931783 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:24.933369 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:24.933373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:24.933382 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:24.933598 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:24.933823 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:24.933824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:24.933825 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:24.933949 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:24.934236 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:24.934240 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:24.934412 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 02:53:24.934550 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 02:53:24.934740 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":154,"threads_started":1}
I20260925 02:53:24.936472 21958 heartbeater.cc:347] Connected to a master server at 127.12.222.254:43351
I20260925 02:53:24.936684 21958 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:24.937083 21958 heartbeater.cc:511] Master 127.12.222.254:43351 requested a full tablet report, sending...
I20260925 02:53:24.938045 21483 ts_manager.cc:194] Registered new tserver with Master: e1f2ca51eb1c4b73b11ea3ef5241c769 (127.12.222.195:45483)
I20260925 02:53:24.938472 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:53:24.938711 21483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:47929
WARNING: no policy specified for oryx/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.193.keytab.
Entry for principal oryx/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.193.keytab.
WARNING: no policy specified for oryx/127.12.222.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.222.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.194.keytab.
Entry for principal oryx/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.194.keytab.
WARNING: no policy specified for oryx/127.12.222.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.222.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.195.keytab.
Entry for principal oryx/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.195.keytab.
I20260925 02:53:25.000437 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 21548
I20260925 02:53:25.005945 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 21688
I20260925 02:53:25.011312 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 21827
I20260925 02:53:25.016175 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 21450
I20260925 02:53:25.021833 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:43351
--webserver_interface=127.12.222.254
--webserver_port=42325
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:36179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:43351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:25.121428 21967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:25.121624 21967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:25.121658 21967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:25.125229 21967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:25.125273 21967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:25.125291 21967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:25.125305 21967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:25.125365 21967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:25.130002 21967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:36179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:43351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:43351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=42325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:25.130788 21967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:25.131666 21967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:25.139122 21975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:25.139117 21973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:25.139117 21972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:25.139297 21967 server_base.cc:1034] running on GCE node
I20260925 02:53:25.139848 21967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:25.140759 21967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:25.141925 21967 hybrid_clock.cc:648] HybridClock initialized: now 1790304805141904 us; error 36 us; skew 500 ppm
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304805, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:25.144459 21967 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:53:25.145213 21967 webserver.cc:460] Webserver started at http://127.12.222.254:42325/ using document root <none> and password file <none>
I20260925 02:53:25.145721 21967 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:25.145814 21967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:25.149127 21967 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:53:25.151026 21982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:25.152087 21967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:25.152218 21967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "bc15923d224e4559b1f509272e0e6c42"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-252r"
server_key: "89d684cbc9caa8758c492de7881683a2"
server_key_iv: "b9192efa5f9bd8ba56f0b8a3180afc26"
server_key_version: "encryptionkey@0"
I20260925 02:53:25.152669 21967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:25.164094 21967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:25.166576 21967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:25.166762 21967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:25.172806 21967 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:43351
I20260925 02:53:25.172829 22034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:43351 every 8 connection(s)
I20260925 02:53:25.173641 21967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:25.176946 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 21967
I20260925 02:53:25.177280 22035 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:53:25.177327 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:43085
--local_ip_for_outbound_sockets=127.12.222.193
--tserver_master_addrs=127.12.222.254:43351
--webserver_port=35739
--webserver_interface=127.12.222.193
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:36179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.222.193
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:53:25.178704 22035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42: Bootstrap starting.
I20260925 02:53:25.193315 22035 log.cc:824] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:25.198885 22035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:53:25.199189 22035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42: Bootstrap complete.
I20260925 02:53:25.202822 22035 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc15923d224e4559b1f509272e0e6c42" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43351 } }
I20260925 02:53:25.203192 22035 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc15923d224e4559b1f509272e0e6c42, State: Initialized, Role: FOLLOWER
I20260925 02:53:25.203661 22035 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc15923d224e4559b1f509272e0e6c42" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43351 } }
I20260925 02:53:25.203795 22035 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:25.203886 22035 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:25.203989 22035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:53:25.205701 22035 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc15923d224e4559b1f509272e0e6c42" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43351 } }
I20260925 02:53:25.206005 22035 leader_election.cc:302] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc15923d224e4559b1f509272e0e6c42; no voters:
I20260925 02:53:25.206271 22035 leader_election.cc:288] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:53:25.206395 22039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:53:25.206633 22039 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [term 2 LEADER]: Becoming Leader. State: Replica: bc15923d224e4559b1f509272e0e6c42, State: Running, Role: LEADER
I20260925 02:53:25.206908 22039 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc15923d224e4559b1f509272e0e6c42" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43351 } }
I20260925 02:53:25.207335 22035 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:25.207764 22039 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc15923d224e4559b1f509272e0e6c42. Latest consensus state: current_term: 2 leader_uuid: "bc15923d224e4559b1f509272e0e6c42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc15923d224e4559b1f509272e0e6c42" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43351 } } }
I20260925 02:53:25.207858 22039 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:25.208096 22041 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bc15923d224e4559b1f509272e0e6c42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc15923d224e4559b1f509272e0e6c42" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43351 } } }
I20260925 02:53:25.208202 22041 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:25.208505 22043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:25.211311 22043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:25.211870 22043 catalog_manager.cc:1259] Loaded cluster ID: b3d6e5dd36d34498bf863d0cfdca9b0e
I20260925 02:53:25.211931 22043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:25.214177 22043 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:25.214748 22043 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bc15923d224e4559b1f509272e0e6c42: Loaded TSK: 0
I20260925 02:53:25.215813 22043 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:53:25.278935 22037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:25.279135 22037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:25.279170 22037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:25.282642 22037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:25.282684 22037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:25.282752 22037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:53:25.287114 22037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:36179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.193.keytab
--principal=oryx/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:43085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=35739
--tserver_master_addrs=127.12.222.254:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:25.287858 22037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:25.288755 22037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:25.297242 22063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:25.297246 22062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:25.297254 22065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:25.297554 22037 server_base.cc:1034] running on GCE node
I20260925 02:53:25.297906 22037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:25.298466 22037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:25.299645 22037 hybrid_clock.cc:648] HybridClock initialized: now 1790304805299623 us; error 39 us; skew 500 ppm
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304805, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:25.302394 22037 init.cc:377] Logged in from keytab as oryx/127.12.222.193@KRBTEST.COM (short username oryx)
I20260925 02:53:25.303171 22037 webserver.cc:460] Webserver started at http://127.12.222.193:35739/ using document root <none> and password file <none>
I20260925 02:53:25.303695 22037 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:25.303794 22037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:25.307626 22037 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:53:25.309717 22072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:25.310895 22037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:53:25.311008 22037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "46ab7dd4116a4780959f9fc0b59d9e8c"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-252r"
server_key: "5edd8751c8643b1dd41f6c51ad09c1ae"
server_key_iv: "bcdee6d067a9987d19689299e28d5c25"
server_key_version: "encryptionkey@0"
I20260925 02:53:25.311468 22037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:25.327278 22037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:25.330240 22037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:25.330437 22037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:25.330986 22037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:25.331766 22037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:25.331809 22037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:25.331843 22037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:25.331888 22037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:25.340508 22037 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:43085
I20260925 02:53:25.340583 22185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:43085 every 8 connection(s)
I20260925 02:53:25.341431 22037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:53:25.342723 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 22037
I20260925 02:53:25.343230 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:43881
--local_ip_for_outbound_sockets=127.12.222.194
--tserver_master_addrs=127.12.222.254:43351
--webserver_port=33407
--webserver_interface=127.12.222.194
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:36179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.222.194
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:53:25.354923 22189 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:25.343368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:44089 (local address 127.12.222.254:43351)
0925 02:53:25.344469 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:25.344487 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:25.344526 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:53:25.345391 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:25.345485 (+ 94us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:25.345827 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:25.346151 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:25.347174 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.348820 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.349889 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.350692 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.350882 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.354321 (+ 3439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.193:44089: BlockingRecv error: recv got EOF from 127.12.222.193:44089 (error 108)
Metrics: {"server-negotiator.queue_time_us":945,"thread_start_us":106,"threads_started":1}
I20260925 02:53:25.354981 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:25.343580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:44089)
0925 02:53:25.344198 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:25.344283 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:25.345026 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:25.346221 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:25.346227 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:25.346532 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:25.346979 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.346992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.348953 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.348956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.349773 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.349781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.350842 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.350844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.351005 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.351591 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:25.351607 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:25.354171 (+ 2564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":128,"threads_started":1}
W20260925 02:53:25.355538 22186 heartbeater.cc:644] Failed to heartbeat to 127.12.222.254:43351 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.222.254:43351: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:25.360633 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:25.355912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:50273)
0925 02:53:25.356023 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:25.356036 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:25.356110 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:25.356337 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:25.356341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:25.356505 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:25.356609 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.356615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.357266 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.357269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.358068 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.358077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.358498 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.358502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.358585 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.358943 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:25.358956 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:25.360533 (+ 1577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 02:53:25.360977 22189 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:25.356011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:50273 (local address 127.12.222.254:43351)
0925 02:53:25.356138 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:25.356141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:25.356152 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:53:25.356179 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:25.356181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:25.356215 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:25.356420 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:25.356722 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.357141 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.358170 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.358403 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.358473 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.360634 (+ 2161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.193:50273: BlockingRecv error: recv got EOF from 127.12.222.193:50273 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:25.365494 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:25.361128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:49005)
0925 02:53:25.361232 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:25.361245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:25.361321 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:25.361547 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:25.361548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:25.361654 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:25.361710 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.361714 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.362584 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.362585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.363022 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.363028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.363526 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.363528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.363576 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.363957 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:25.363967 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:25.365410 (+ 1443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 02:53:25.365680 22189 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:25.361245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:49005 (local address 127.12.222.254:43351)
0925 02:53:25.361364 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:25.361368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:25.361380 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:53:25.361410 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:25.361412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:25.361454 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:25.361525 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:25.361801 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.362492 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.363123 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.363432 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.363524 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.365584 (+ 2060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.193:49005: BlockingRecv error: recv got EOF from 127.12.222.193:49005 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 02:53:25.365675 22186 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:53:25.446682 22191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:25.446870 22191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:25.446905 22191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:25.450487 22191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:25.450531 22191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:25.450579 22191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:53:25.455056 22191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:36179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.194.keytab
--principal=oryx/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:43881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=33407
--tserver_master_addrs=127.12.222.254:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:25.455971 22191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:25.456987 22191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:53:25.464974 22191 server_base.cc:1034] running on GCE node
W20260925 02:53:25.464931 22199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:25.464903 22197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:25.464903 22196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:25.465494 22191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:25.466040 22191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:25.467218 22191 hybrid_clock.cc:648] HybridClock initialized: now 1790304805467205 us; error 32 us; skew 500 ppm
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304805, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:25.469594 22191 init.cc:377] Logged in from keytab as oryx/127.12.222.194@KRBTEST.COM (short username oryx)
I20260925 02:53:25.470291 22191 webserver.cc:460] Webserver started at http://127.12.222.194:33407/ using document root <none> and password file <none>
I20260925 02:53:25.470773 22191 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:25.470873 22191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:25.474202 22191 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260925 02:53:25.475965 22206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:25.477080 22191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:25.477216 22191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "0f65e960d86445fcae3b3ecf1e0113d4"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-252r"
server_key: "6b28d0e189416acd71f50a94c66f45a9"
server_key_iv: "5c84cbd34d571f4ddbc157f107b4a31f"
server_key_version: "encryptionkey@0"
I20260925 02:53:25.477687 22191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:25.489342 22191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:25.491549 22191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:25.491742 22191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:25.492190 22191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:25.492909 22191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:25.492970 22191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:25.493031 22191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:25.493067 22191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:25.500820 22191 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:43881
I20260925 02:53:25.500844 22319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:43881 every 8 connection(s)
I20260925 02:53:25.501633 22191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:25.509640 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 22191
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:25.510126 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:45483
--local_ip_for_outbound_sockets=127.12.222.195
--tserver_master_addrs=127.12.222.254:43351
--webserver_port=40517
--webserver_interface=127.12.222.195
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:36179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.222.195
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:25.510885 22189 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:25.503083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:37925 (local address 127.12.222.254:43351)
0925 02:53:25.503198 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:25.503201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:25.504003 (+ 802us) server_negotiation.cc:407] Connection header received
0925 02:53:25.504696 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:25.504698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:25.504742 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:25.504806 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:25.505753 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.506239 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.507331 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.507525 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.507612 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.510772 (+ 3160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.194:37925: BlockingRecv error: recv got EOF from 127.12.222.194:37925 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:53:25.511376 22322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:25.503355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:37925)
0925 02:53:25.503872 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:25.503904 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:25.504541 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:25.504937 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:25.504942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:25.505233 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:25.505639 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.505650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.506339 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.506341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.507224 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.507232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.507686 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.507689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.507806 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.508330 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:25.508344 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:25.510655 (+ 2311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":100,"threads_started":1}
W20260925 02:53:25.511775 22320 heartbeater.cc:644] Failed to heartbeat to 127.12.222.254:43351 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.222.254:43351: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:25.517231 22322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:25.512159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:36723)
0925 02:53:25.512271 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:25.512282 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:25.512351 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:25.512575 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:25.512577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:25.512711 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:25.512787 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.512792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.513416 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.513420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.513889 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.513896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.514538 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.514540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.514601 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.515051 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:25.515070 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:25.517058 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 02:53:25.517246 22189 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:25.512270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:36723 (local address 127.12.222.254:43351)
0925 02:53:25.512393 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:25.512398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:25.512410 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:53:25.512448 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:25.512450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:25.512488 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:25.512561 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:25.512889 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.513316 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.514060 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.514432 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.514553 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.517157 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.194:36723: BlockingRecv error: recv got EOF from 127.12.222.194:36723 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:25.522258 22322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:25.517806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:57603)
0925 02:53:25.517888 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:25.517901 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:25.517995 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:25.518183 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:25.518186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:25.518315 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:25.518370 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.518374 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.519077 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.519081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.519579 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.519586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.519999 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.520002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.520074 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.520510 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:25.520525 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:25.522136 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 02:53:25.522336 22189 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:25.517890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:57603 (local address 127.12.222.254:43351)
0925 02:53:25.518002 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:25.518006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:25.518019 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:53:25.518057 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:25.518061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:25.518098 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:25.518170 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:25.518475 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.518963 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.519691 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.519891 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.519996 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.522233 (+ 2237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.194:57603: BlockingRecv error: recv got EOF from 127.12.222.194:57603 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 02:53:25.522539 22320 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:53:25.612262 22323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:25.612460 22323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:25.612526 22323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:25.616181 22323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:25.616266 22323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:25.616389 22323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:53:25.620857 22323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:36179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/krb5kdc/oryx_127.12.222.195.keytab
--principal=oryx/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:45483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=40517
--tserver_master_addrs=127.12.222.254:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:25.621692 22323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:25.622550 22323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:25.630891 22331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:25.630949 22329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:25.631011 22323 server_base.cc:1034] running on GCE node
W20260925 02:53:25.630893 22328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:25.631405 22323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:25.631876 22323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:25.633033 22323 hybrid_clock.cc:648] HybridClock initialized: now 1790304805633017 us; error 30 us; skew 500 ppm
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304805, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:25.635630 22323 init.cc:377] Logged in from keytab as oryx/127.12.222.195@KRBTEST.COM (short username oryx)
I20260925 02:53:25.636413 22323 webserver.cc:460] Webserver started at http://127.12.222.195:40517/ using document root <none> and password file <none>
I20260925 02:53:25.636896 22323 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:25.636986 22323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:25.640080 22323 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:53:25.641860 22338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:25.642921 22323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:53:25.643039 22323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "e1f2ca51eb1c4b73b11ea3ef5241c769"
format_stamp: "Formatted at 2026-09-25 02:53:24 on dist-test-slave-252r"
server_key: "c6ca1b4966f30335f3b265ce6be396c3"
server_key_iv: "5d5c3178152b990abce2b634fc261fb8"
server_key_version: "encryptionkey@0"
I20260925 02:53:25.643482 22323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:25.675233 22323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:25.678139 22323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:25.678308 22323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:25.678797 22323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:25.679497 22323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:25.679562 22323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:25.679620 22323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:25.679663 22323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:25.687608 22323 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:45483
I20260925 02:53:25.687629 22453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:45483 every 8 connection(s)
I20260925 02:53:25.688473 22323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:25.696466 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 22323
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:53:25.698124 22189 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:25.689963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:39215 (local address 127.12.222.254:43351)
0925 02:53:25.690082 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:25.690087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:25.690966 (+ 879us) server_negotiation.cc:407] Connection header received
0925 02:53:25.691787 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:25.691790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:25.691845 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:25.691931 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:25.692991 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.693463 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.694470 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.694662 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.694760 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.697992 (+ 3232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.195:39215: BlockingRecv error: recv got EOF from 127.12.222.195:39215 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:53:25.698570 22456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:25.690239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.195:39215)
0925 02:53:25.690822 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:25.690858 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:25.691593 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:25.692092 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:25.692098 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:25.692459 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:25.692852 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.692863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.693584 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.693588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.694337 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.694345 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.694793 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.694797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.694948 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.695433 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:25.695466 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:25.697856 (+ 2390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":123,"threads_started":1}
W20260925 02:53:25.698966 22454 heartbeater.cc:644] Failed to heartbeat to 127.12.222.254:43351 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.222.254:43351: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:25.704624 22456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:25.699310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.195:43111)
0925 02:53:25.699517 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:25.699529 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:25.699613 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:25.699807 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:25.699810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:25.699947 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:25.700043 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.700050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.700890 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.700894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.701421 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.701430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.701994 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.702036 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.702167 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.702587 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:25.702600 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:25.704508 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146}
W20260925 02:53:25.704906 22189 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:25.699378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:43111 (local address 127.12.222.254:43351)
0925 02:53:25.699485 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:25.699488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:25.699581 (+ 93us) server_negotiation.cc:407] Connection header received
0925 02:53:25.699678 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:25.699681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:25.699713 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:25.699794 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:25.700155 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.700773 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.701554 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.701883 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.701977 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.704641 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.195:43111: BlockingRecv error: recv got EOF from 127.12.222.195:43111 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:53:25.706780 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:25.697409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53496 (local address 127.12.222.254:43351)
0925 02:53:25.697655 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:25.697659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:25.697672 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:53:25.697704 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:25.697707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:25.697743 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:25.697835 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:25.698617 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.699256 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.700072 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.700391 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.700465 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.701775 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:25.701782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:25.701797 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:25.701831 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:25.704174 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:25.705255 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:25.705259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:25.705263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:25.705397 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:25.705683 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:25.705687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:25.705688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:25.705996 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:25.706186 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:25.706461 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 02:53:25.706688 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:25 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:25.710026 22456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:25.705088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.195:39397)
0925 02:53:25.705175 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:25.705185 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:25.705261 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:25.705658 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:25.705660 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:25.705804 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:25.705862 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.705866 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.706521 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.706527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.707183 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:25.707191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.707691 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.707694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:25.707768 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.708332 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:25.708344 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:25.709955 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 02:53:25.710137 22189 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:25.705290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:39397 (local address 127.12.222.254:43351)
0925 02:53:25.705465 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:25.705470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:25.705485 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:53:25.705518 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:25.705520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:25.705552 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:25.705642 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:25.705971 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.706419 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.707295 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:25.707586 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:25.707679 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:25.710050 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.195:39397: BlockingRecv error: recv got EOF from 127.12.222.195:39397 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260925 02:53:25.710213 22454 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 02:53:26 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:26 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:26.371493 22459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:26.366256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:44611)
0925 02:53:26.366545 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:26.366562 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:26.366683 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:26.366941 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:26.366944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:26.367137 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:26.367252 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:26.367257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.368000 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.368005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:26.368599 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:26.368607 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.369079 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.369082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:26.369172 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:26.369612 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:26.369622 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:26.371390 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":79,"threads_started":1}
W20260925 02:53:26.371670 22460 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:26.366285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:44611 (local address 127.12.222.254:43351)
0925 02:53:26.366544 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:26.366549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:26.366651 (+ 102us) server_negotiation.cc:407] Connection header received
0925 02:53:26.366778 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:26.366782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:26.366832 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:26.366927 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:26.367360 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.367862 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.368721 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.368955 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.369061 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:26.371541 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.193:44611: BlockingRecv error: recv got EOF from 127.12.222.193:44611 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":83,"threads_started":1}
Sep 25 02:53:26 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:26 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:26.528492 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:26.523111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:44319)
0925 02:53:26.523418 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:26.523433 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:26.523560 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:26.523844 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:26.523847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:26.524065 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:26.524165 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:26.524170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.524908 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.524912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:26.525441 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:26.525449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.525945 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.525948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:26.526044 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:26.526540 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:26.526551 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:26.528342 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":79,"threads_started":1}
W20260925 02:53:26.528582 22460 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:26.523225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:44319 (local address 127.12.222.254:43351)
0925 02:53:26.523394 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:26.523398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:26.523512 (+ 114us) server_negotiation.cc:407] Connection header received
0925 02:53:26.523676 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:26.523680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:26.523730 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:26.523816 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:26.524303 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.524795 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.525582 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.525801 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.525930 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:26.528466 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.194:44319: BlockingRecv error: recv got EOF from 127.12.222.194:44319 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 02:53:26 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:26 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:26.716081 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:26.710738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.195:57139)
0925 02:53:26.711017 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:26.711028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:26.711136 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:26.711397 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:26.711400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:26.711592 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:26.711696 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:26.711701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.712469 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.712474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:26.713011 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:26.713018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.713466 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.713469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:26.713561 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:26.714109 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:26.714123 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:26.715914 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":92,"threads_started":1}
W20260925 02:53:26.716120 22460 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:26.710775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:57139 (local address 127.12.222.254:43351)
0925 02:53:26.710929 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:26.710934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:26.711090 (+ 156us) server_negotiation.cc:407] Connection header received
0925 02:53:26.711239 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:26.711242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:26.711299 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:26.711379 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:26.711806 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.712367 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.713125 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:26.713354 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:26.713459 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:26.716026 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.195:57139: BlockingRecv error: recv got EOF from 127.12.222.195:57139 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 02:53:27 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:27 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:27.378082 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:27.372328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:44853)
0925 02:53:27.372621 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:27.372636 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:27.372760 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:27.373100 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:27.373103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:27.373307 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:27.373402 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:27.373429 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:27.374247 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:27.374250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:27.374758 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:27.374764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:27.375139 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:27.375141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:27.375194 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:27.375780 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:27.375803 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:27.377956 (+ 2153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":66,"threads_started":1}
W20260925 02:53:27.378270 22464 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:27.372540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:44853 (local address 127.12.222.254:43351)
0925 02:53:27.372752 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:27.372755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:27.372765 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:53:27.372868 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:27.372870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:27.372928 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:27.373013 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:27.373586 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:27.374144 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:27.374850 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:27.375047 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:27.375114 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:27.378094 (+ 2980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.193:44853: BlockingRecv error: recv got EOF from 127.12.222.193:44853 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
Sep 25 02:53:27 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:27 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:27.534518 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:27.529303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:50235)
0925 02:53:27.529549 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:27.529561 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:27.529650 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:27.529899 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:27.529901 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:27.530130 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:27.530243 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:27.530249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:27.531004 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:27.531009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:27.531576 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:27.531584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:27.532052 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:27.532056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:27.532140 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:27.532672 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:27.532683 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:27.534363 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":64,"threads_started":1}
W20260925 02:53:27.534610 22464 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:27.529389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:50235 (local address 127.12.222.254:43351)
0925 02:53:27.529561 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:27.529564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:27.529629 (+ 65us) server_negotiation.cc:407] Connection header received
0925 02:53:27.529729 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:27.529731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:27.529794 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:27.529874 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:27.530366 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:27.530887 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:27.531706 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:27.531934 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:27.532048 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:27.534502 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.194:50235: BlockingRecv error: recv got EOF from 127.12.222.194:50235 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 02:53:27 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:27 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:27.724668 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:27.716825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.195:54285)
0925 02:53:27.717071 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:27.717082 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:27.717165 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:27.717334 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:27.717336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:27.717491 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:27.717579 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:27.717584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:27.718262 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:27.718264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:27.718760 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:27.718767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:27.719138 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:27.719140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:27.719198 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:27.719731 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:27.719746 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:27.724487 (+ 4741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":79,"threads_started":1}
W20260925 02:53:27.724859 22464 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:27.716968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:54285 (local address 127.12.222.254:43351)
0925 02:53:27.717086 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:27.717090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:27.717125 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:53:27.717224 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:27.717227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:27.717264 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:27.717320 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:27.717671 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:27.718172 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:27.718849 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:27.719049 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:27.719117 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:27.724662 (+ 5545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.195:54285: BlockingRecv error: recv got EOF from 127.12.222.195:54285 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 02:53:28 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:28 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:28.389806 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:28.378894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:51463)
0925 02:53:28.379150 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:28.379162 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:28.379243 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:28.379446 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:28.379448 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:28.379604 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:28.379692 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:28.379697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.380419 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.380421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:28.381035 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:28.381045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.381446 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.381449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:28.381506 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:28.387999 (+ 6493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:28.388010 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:28.389692 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":93,"threads_started":1}
W20260925 02:53:28.389979 22468 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:28.379012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:51463 (local address 127.12.222.254:43351)
0925 02:53:28.379236 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:28.379239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:28.379252 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:53:28.379323 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:28.379325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:28.379364 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:28.379438 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:28.379789 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.380332 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.381135 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.381349 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.381428 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:28.389814 (+ 8386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.193:51463: BlockingRecv error: recv got EOF from 127.12.222.193:51463 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
Sep 25 02:53:28 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:28 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:28.540776 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:28.535334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:59845)
0925 02:53:28.535637 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:28.535652 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:28.535779 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:28.536058 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:28.536061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:28.536290 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:28.536384 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:28.536389 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.537149 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.537153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:28.537732 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:28.537740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.538185 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.538188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:28.538273 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:28.538872 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:28.538883 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:28.540665 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":85,"threads_started":1}
W20260925 02:53:28.540971 22468 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:28.535512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:59845 (local address 127.12.222.254:43351)
0925 02:53:28.535639 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:28.535643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:28.535725 (+ 82us) server_negotiation.cc:407] Connection header received
0925 02:53:28.535890 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:28.535894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:28.535951 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:28.536034 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:28.536504 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.537029 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.537857 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.538078 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.538181 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:28.540787 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.194:59845: BlockingRecv error: recv got EOF from 127.12.222.194:59845 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 02:53:28 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:28 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:28.730082 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:28.725310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.195:44261)
0925 02:53:28.725592 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:28.725604 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:28.725700 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:28.725926 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:28.725928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:28.726104 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:28.726197 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:28.726201 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.726862 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.726865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:28.727393 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:28.727400 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.727757 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.727759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:28.727816 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:28.728273 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:28.728294 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:28.729967 (+ 1673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":115,"threads_started":1}
W20260925 02:53:28.730183 22468 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:28.725353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:44261 (local address 127.12.222.254:43351)
0925 02:53:28.725490 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:28.725495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:28.725671 (+ 176us) server_negotiation.cc:407] Connection header received
0925 02:53:28.725763 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:28.725765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:28.725810 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:28.725890 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:28.726289 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.726779 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.727483 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:28.727671 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:28.727738 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:28.730079 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.195:44261: BlockingRecv error: recv got EOF from 127.12.222.195:44261 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 02:53:29 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:29 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:29.396317 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.390592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:41681)
0925 02:53:29.390945 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:29.390958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:29.391045 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:29.391253 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:29.391255 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:29.391417 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:29.391512 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.391517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.392417 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.392420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.393295 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.393306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.393792 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.393795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.393878 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.394368 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:29.394381 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:29.396123 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":129,"threads_started":1}
W20260925 02:53:29.396395 22472 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:29.390789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:41681 (local address 127.12.222.254:43351)
0925 02:53:29.391075 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:29.391078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:29.391091 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:53:29.391126 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:29.391128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:29.391167 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:29.391244 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:29.391621 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.392309 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.393421 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.393673 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.393785 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.396270 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.193:41681: BlockingRecv error: recv got EOF from 127.12.222.193:41681 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":134,"threads_started":1}
Sep 25 02:53:29 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:29 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:29.546768 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.541477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:35253)
0925 02:53:29.541792 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:29.541804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:29.541943 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:29.542221 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:29.542223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:29.542431 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:29.542529 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.542534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.543275 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.543279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.543831 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.543838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.544299 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.544302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.544381 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.544881 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:29.544900 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:29.546661 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":82,"threads_started":1}
W20260925 02:53:29.546900 22472 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:29.541536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:35253 (local address 127.12.222.254:43351)
0925 02:53:29.541709 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:29.541713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:29.541889 (+ 176us) server_negotiation.cc:407] Connection header received
0925 02:53:29.542046 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:29.542051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:29.542115 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:29.542200 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:29.542640 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.543152 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.543960 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.544155 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.544284 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.546785 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.194:35253: BlockingRecv error: recv got EOF from 127.12.222.194:35253 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 02:53:29 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:29 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:29.736193 22474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:29.730923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.195:50117)
0925 02:53:29.731191 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:29.731207 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:29.731345 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:29.731597 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:29.731600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:29.731809 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:29.731906 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.731911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.732712 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.732715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.733251 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:29.733259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.733695 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.733699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:29.733781 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.734289 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:29.734300 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:29.736077 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":85,"threads_started":1}
W20260925 02:53:29.736384 22472 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:29.731011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:50117 (local address 127.12.222.254:43351)
0925 02:53:29.731151 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:29.731156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:29.731275 (+ 119us) server_negotiation.cc:407] Connection header received
0925 02:53:29.731450 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:29.731452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:29.731495 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:29.731577 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:29.732029 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.732602 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.733374 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:29.733578 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:29.733690 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:29.736265 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.195:50117: BlockingRecv error: recv got EOF from 127.12.222.195:50117 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 02:53:30 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:30 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:30.402151 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:30.397096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:48801)
0925 02:53:30.397394 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:30.397407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:30.397497 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:30.397719 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:30.397721 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:30.397888 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:30.397978 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:30.397983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:30.398638 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:30.398640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:30.399150 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:30.399156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:30.399534 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:30.399537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:30.399592 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:30.400041 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:30.400052 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:30.402050 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":94,"threads_started":1}
W20260925 02:53:30.402323 22476 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:30.397219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:48801 (local address 127.12.222.254:43351)
0925 02:53:30.397443 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:30.397447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:30.397459 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:53:30.397575 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:30.397577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:30.397616 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:30.397684 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:30.398073 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:30.398552 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:30.399247 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:30.399446 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:30.399512 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:30.402169 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.193:48801: BlockingRecv error: recv got EOF from 127.12.222.193:48801 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":55,"threads_started":1}
Sep 25 02:53:30 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:30 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:30.552572 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:30.547605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:50017)
0925 02:53:30.547934 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:30.547952 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:30.548065 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:30.548256 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:30.548258 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:30.548416 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:30.548512 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:30.548516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:30.549165 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:30.549167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:30.549736 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:30.549744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:30.550151 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:30.550154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:30.550211 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:30.550665 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:30.550676 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:30.552464 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":126,"threads_started":1}
W20260925 02:53:30.552703 22476 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:30.547716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:50017 (local address 127.12.222.254:43351)
0925 02:53:30.547864 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:30.547870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:30.548042 (+ 172us) server_negotiation.cc:407] Connection header received
0925 02:53:30.548127 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:30.548129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:30.548168 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:30.548259 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:30.548600 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:30.549079 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:30.549835 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:30.550055 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:30.550129 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:30.552589 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.194:50017: BlockingRecv error: recv got EOF from 127.12.222.194:50017 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 02:53:30 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:30 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:30.742445 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:30.736991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.195:53433)
0925 02:53:30.737248 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:30.737261 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:30.737348 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:30.737607 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:30.737609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:30.737788 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:30.737885 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:30.737890 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:30.738771 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:30.738775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:30.739534 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:30.739547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:30.740028 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:30.740032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:30.740117 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:30.740586 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:30.740596 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:30.742280 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
W20260925 02:53:30.742511 22476 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:30.737089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:53433 (local address 127.12.222.254:43351)
0925 02:53:30.737219 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:30.737223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:30.737325 (+ 102us) server_negotiation.cc:407] Connection header received
0925 02:53:30.737427 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:30.737430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:30.737473 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:30.737551 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:30.737993 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:30.738624 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:30.739654 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:30.739910 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:30.740021 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:30.742396 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.195:53433: BlockingRecv error: recv got EOF from 127.12.222.195:53433 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 02:53:31 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:31 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:31.408167 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:31.402894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:39651)
0925 02:53:31.403237 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:31.403258 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:31.403344 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:31.403555 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:31.403557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:31.403719 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:31.403806 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:31.403811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:31.404564 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:31.404566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:31.405068 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:31.405075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:31.405435 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:31.405437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:31.405492 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:31.405939 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:31.405950 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:31.407980 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":105,"threads_started":1}
W20260925 02:53:31.408277 22480 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:31.403122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:39651 (local address 127.12.222.254:43351)
0925 02:53:31.403339 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:31.403342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:31.403354 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:53:31.403406 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:31.403408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:31.403447 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:31.403516 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:31.403895 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:31.404475 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:31.405154 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:31.405349 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:31.405415 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:31.408126 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.193:39651: BlockingRecv error: recv got EOF from 127.12.222.193:39651 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":66,"threads_started":1}
Sep 25 02:53:31 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:31 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:31.558880 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:31.553342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:42177)
0925 02:53:31.553691 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:31.553702 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:31.553834 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:31.554206 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:31.554209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:31.554436 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:31.554562 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:31.554570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:31.555404 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:31.555409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:31.556015 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:31.556025 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:31.556614 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:31.556618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:31.556712 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:31.557173 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:31.557183 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:31.558780 (+ 1597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":198,"threads_started":1}
W20260925 02:53:31.559060 22480 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:31.553528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:42177 (local address 127.12.222.254:43351)
0925 02:53:31.553651 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:31.553654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:31.553772 (+ 118us) server_negotiation.cc:407] Connection header received
0925 02:53:31.553956 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:31.553974 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:31.554050 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:31.554145 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:31.554687 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:31.555284 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:31.556138 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:31.556491 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:31.556599 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:31.558900 (+ 2301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.194:42177: BlockingRecv error: recv got EOF from 127.12.222.194:42177 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 02:53:31 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:31 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:31.748871 22482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:31.743499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.195:57879)
0925 02:53:31.743803 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:31.743821 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:31.743947 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:31.744190 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:31.744192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:31.744407 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:31.744503 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:31.744508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:31.745216 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:31.745218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:31.745740 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:31.745746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:31.746231 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:31.746234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:31.746299 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:31.746751 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:31.746762 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:31.748714 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":105,"threads_started":1}
W20260925 02:53:31.748912 22480 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:31.743552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:57879 (local address 127.12.222.254:43351)
0925 02:53:31.743689 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:31.743693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:31.743896 (+ 203us) server_negotiation.cc:407] Connection header received
0925 02:53:31.744047 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:31.744049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:31.744093 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:31.744150 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:31.744599 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:31.745130 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:31.745878 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:31.746135 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:31.746211 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:31.748821 (+ 2610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.195:57879: BlockingRecv error: recv got EOF from 127.12.222.195:57879 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 02:53:32 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:32 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:32.413964 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:32.408862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:56657)
0925 02:53:32.409177 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:32.409190 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:32.409280 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:32.409536 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:32.409538 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:32.409715 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:32.409800 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:32.409805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:32.410462 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:32.410464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:32.410957 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:32.410963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:32.411332 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:32.411335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:32.411387 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:32.411935 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:32.411947 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:32.413849 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":85,"threads_started":1}
W20260925 02:53:32.414136 22484 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:32.408933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:56657 (local address 127.12.222.254:43351)
0925 02:53:32.409207 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:32.409210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:32.409236 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:53:32.409369 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:32.409371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:32.409420 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:32.409502 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:32.409888 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:32.410375 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:32.411045 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:32.411244 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:32.411312 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:32.413966 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.193:56657: BlockingRecv error: recv got EOF from 127.12.222.193:56657 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":76,"threads_started":1}
Sep 25 02:53:32 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:32 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:32.564872 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:32.559619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:34813)
0925 02:53:32.559952 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:32.559964 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:32.560056 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:32.560399 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:32.560401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:32.560612 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:32.560732 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:32.560756 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:32.561584 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:32.561587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:32.562132 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:32.562151 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:32.562515 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:32.562517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:32.562573 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:32.563013 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:32.563023 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:32.564754 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":71,"threads_started":1}
W20260925 02:53:32.565040 22484 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:32.559709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:34813 (local address 127.12.222.254:43351)
0925 02:53:32.559846 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:32.559849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:32.560017 (+ 168us) server_negotiation.cc:407] Connection header received
0925 02:53:32.560171 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:32.560173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:32.560264 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:32.560342 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:32.560904 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:32.561484 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:32.562237 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:32.562429 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:32.562497 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:32.564890 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.194:34813: BlockingRecv error: recv got EOF from 127.12.222.194:34813 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 02:53:32 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:32 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:32.754879 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:32.749513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.195:43759)
0925 02:53:32.749809 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:32.749821 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:32.749916 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:32.750189 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:32.750192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:32.750455 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:32.750605 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:32.750611 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:32.751573 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:32.751576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:32.752130 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:32.752138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:32.752539 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:32.752541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:32.752601 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:32.753056 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:32.753067 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:32.754768 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":96,"threads_started":1}
W20260925 02:53:32.754988 22484 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:32.749578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:43759 (local address 127.12.222.254:43351)
0925 02:53:32.749710 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:32.749713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:32.749896 (+ 183us) server_negotiation.cc:407] Connection header received
0925 02:53:32.749996 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:32.749999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:32.750055 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:32.750150 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:32.750750 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:32.751475 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:32.752244 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:32.752449 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:32.752522 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:32.754893 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.195:43759: BlockingRecv error: recv got EOF from 127.12.222.195:43759 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 02:53:33 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:33 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:33.424607 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:33.415029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:41537)
0925 02:53:33.415330 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:33.415345 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:33.415457 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:33.415780 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:33.415784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:33.416023 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:33.416145 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:33.416152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:33.416989 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:33.416992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:33.417527 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:33.417533 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:33.417913 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:33.417915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:33.417979 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:33.422689 (+ 4710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:33.422700 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:33.424442 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
W20260925 02:53:33.424661 22488 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:33.415147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:41537 (local address 127.12.222.254:43351)
0925 02:53:33.415355 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:33.415361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:33.415411 (+ 50us) server_negotiation.cc:407] Connection header received
0925 02:53:33.415569 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:33.415575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:33.415630 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:33.415733 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:33.416323 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:33.416869 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:33.417630 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:33.417826 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:33.417892 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:33.424552 (+ 6660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.193:41537: BlockingRecv error: recv got EOF from 127.12.222.193:41537 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":54,"threads_started":1}
Sep 25 02:53:33 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:33 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:33.570570 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:33.565517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:52155)
0925 02:53:33.565759 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:33.565770 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:33.565879 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:33.566125 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:33.566128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:33.566313 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:33.566406 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:33.566411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:33.567275 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:33.567277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:33.567825 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:33.567832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:33.568219 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:33.568221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:33.568340 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:33.568786 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:33.568796 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:33.570462 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
W20260925 02:53:33.570701 22488 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:33.565584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:52155 (local address 127.12.222.254:43351)
0925 02:53:33.565755 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:33.565759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:33.565815 (+ 56us) server_negotiation.cc:407] Connection header received
0925 02:53:33.565961 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:33.565963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:33.566003 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:33.566074 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:33.566524 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:33.567183 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:33.567919 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:33.568134 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:33.568200 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:33.570579 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.194:52155: BlockingRecv error: recv got EOF from 127.12.222.194:52155 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 02:53:33 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:33 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:33.760967 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:33.755787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.195:34693)
0925 02:53:33.756074 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:33.756088 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:33.756201 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:33.756531 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:33.756533 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:33.756715 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:33.756817 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:33.756822 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:33.757585 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:33.757587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:33.758087 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:33.758094 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:33.758543 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:33.758545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:33.758599 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:33.759020 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:33.759030 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:33.760824 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":99,"threads_started":1}
W20260925 02:53:33.761129 22488 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:33.755822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:34693 (local address 127.12.222.254:43351)
0925 02:53:33.755969 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:33.755972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:33.756157 (+ 185us) server_negotiation.cc:407] Connection header received
0925 02:53:33.756334 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:33.756340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:33.756397 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:33.756478 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:33.756922 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:33.757498 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:33.758176 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:33.758449 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:33.758522 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:33.760981 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.195:34693: BlockingRecv error: recv got EOF from 127.12.222.195:34693 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 02:53:34 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:34 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:34.430343 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.425323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:60233)
0925 02:53:34.425608 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.425620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:34.425705 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:34.425946 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:34.425948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:34.426111 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:34.426203 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.426212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.426978 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.426980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.427523 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.427530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.427910 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.427912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.427977 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.428462 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:34.428473 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:34.430242 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
W20260925 02:53:34.430487 22492 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:34.425434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:60233 (local address 127.12.222.254:43351)
0925 02:53:34.425664 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:34.425668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:34.425680 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:53:34.425791 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:34.425793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:34.425836 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:34.425907 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:34.426331 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.426875 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.427613 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.427821 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.427892 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.430360 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.193:60233: BlockingRecv error: recv got EOF from 127.12.222.193:60233 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":58,"threads_started":1}
Sep 25 02:53:34 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:34 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:34.576391 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.571367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:51405)
0925 02:53:34.571636 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.571650 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:34.571769 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:34.572081 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:34.572084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:34.572278 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:34.572395 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.572400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.573033 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.573035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.573565 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.573572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.573952 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.573954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.574013 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.574452 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:34.574461 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:34.576170 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":103,"threads_started":1}
W20260925 02:53:34.576550 22492 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:34.571430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:51405 (local address 127.12.222.254:43351)
0925 02:53:34.571593 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:34.571598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:34.571725 (+ 127us) server_negotiation.cc:407] Connection header received
0925 02:53:34.571884 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:34.571888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:34.571948 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:34.572029 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:34.572484 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.572949 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.573656 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.573855 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.573925 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.576399 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.194:51405: BlockingRecv error: recv got EOF from 127.12.222.194:51405 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 02:53:34 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:34 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:34.766964 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:34.761704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.195:59247)
0925 02:53:34.762111 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:34.762125 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:34.762238 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:34.762506 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:34.762509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:34.762717 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:34.762800 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.762804 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.763525 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.763530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.764064 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:34.764072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.764562 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.764565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:34.764647 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.765089 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:34.765099 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:34.766794 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":105,"threads_started":1}
W20260925 02:53:34.767004 22492 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:34.761777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:59247 (local address 127.12.222.254:43351)
0925 02:53:34.761917 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:34.761924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:34.762225 (+ 301us) server_negotiation.cc:407] Connection header received
0925 02:53:34.762320 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:34.762323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:34.762389 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:34.762476 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:34.762914 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.763423 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.764186 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:34.764456 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:34.764558 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:34.766908 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.195:59247: BlockingRecv error: recv got EOF from 127.12.222.195:59247 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 02:53:35 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:35 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:35.436172 22495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:35.431049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:33323)
0925 02:53:35.431322 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:35.431335 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:35.431420 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:35.431667 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:35.431669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:35.431840 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:35.431933 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:35.431939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:35.432711 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:35.432713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:35.433211 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:35.433218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:35.433595 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:35.433598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:35.433652 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:35.434275 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:35.434286 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:35.436069 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":65,"threads_started":1}
W20260925 02:53:35.436321 22496 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:35.431132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:33323 (local address 127.12.222.254:43351)
0925 02:53:35.431397 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:35.431400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:35.431412 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:53:35.431513 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:35.431515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:35.431556 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:35.431628 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:35.432031 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:35.432621 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:35.433303 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:35.433506 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:35.433574 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:35.436178 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.193:33323: BlockingRecv error: recv got EOF from 127.12.222.193:33323 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":87,"threads_started":1}
Sep 25 02:53:35 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:35 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:35.582492 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:35.577223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:48845)
0925 02:53:35.577525 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:35.577541 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:35.577663 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:35.577959 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:35.577962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:35.578130 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:35.578223 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:35.578228 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:35.578905 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:35.578907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:35.579482 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:35.579490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:35.579897 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:35.579900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:35.579960 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:35.580555 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:35.580574 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:35.582372 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":92,"threads_started":1}
W20260925 02:53:35.582602 22496 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:35.577293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:48845 (local address 127.12.222.254:43351)
0925 02:53:35.577478 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:35.577483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:35.577632 (+ 149us) server_negotiation.cc:407] Connection header received
0925 02:53:35.577772 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:35.577776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:35.577832 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:35.577915 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:35.578318 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:35.578816 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:35.579585 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:35.579801 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:35.579876 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:35.582481 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.194:48845: BlockingRecv error: recv got EOF from 127.12.222.194:48845 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 02:53:35 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:35 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:35.773185 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:35.767737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.195:47285)
0925 02:53:35.768100 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:35.768115 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:35.768251 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:35.768542 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:35.768545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:35.768750 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:35.768837 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:35.768842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:35.769544 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:35.769550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:35.770069 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:35.770076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:35.770504 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:35.770507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:35.770587 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:35.771201 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:35.771212 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:35.773065 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":179,"threads_started":1}
W20260925 02:53:35.773303 22496 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:35.767947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:47285 (local address 127.12.222.254:43351)
0925 02:53:35.768100 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:35.768104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:35.768191 (+ 87us) server_negotiation.cc:407] Connection header received
0925 02:53:35.768377 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:35.768380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:35.768434 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:35.768515 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:35.768960 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:35.769443 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:35.770184 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:35.770393 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:35.770490 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:35.773191 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.195:47285: BlockingRecv error: recv got EOF from 127.12.222.195:47285 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 02:53:36 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:36 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:36.442248 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:36.437060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:42149)
0925 02:53:36.437341 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:36.437353 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:36.437463 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:36.437799 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:36.437802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:36.437995 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:36.438126 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:36.438135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:36.439078 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:36.439081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:36.439582 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:36.439589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:36.439987 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:36.439989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:36.440044 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:36.440523 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:36.440533 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:36.442126 (+ 1593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":88,"threads_started":1}
W20260925 02:53:36.442377 22500 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:36.437125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:42149 (local address 127.12.222.254:43351)
0925 02:53:36.437357 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:36.437360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:36.437420 (+ 60us) server_negotiation.cc:407] Connection header received
0925 02:53:36.437577 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:36.437580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:36.437645 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:36.437722 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:36.438263 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:36.438985 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:36.439685 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:36.439891 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:36.439963 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:36.442249 (+ 2286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.193:42149: BlockingRecv error: recv got EOF from 127.12.222.193:42149 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":83,"threads_started":1}
Sep 25 02:53:36 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:36 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:36.588209 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:36.583144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:39941)
0925 02:53:36.583406 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:36.583419 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:36.583510 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:36.583789 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:36.583792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:36.583993 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:36.584114 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:36.584123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:36.584978 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:36.584980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:36.585484 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:36.585490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:36.585854 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:36.585856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:36.585912 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:36.586362 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:36.586372 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:36.588099 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
W20260925 02:53:36.588430 22500 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:36.583182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:39941 (local address 127.12.222.254:43351)
0925 02:53:36.583315 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:36.583318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:36.583482 (+ 164us) server_negotiation.cc:407] Connection header received
0925 02:53:36.583598 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:36.583601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:36.583655 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:36.583740 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:36.584253 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:36.584881 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:36.585574 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:36.585766 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:36.585833 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:36.588278 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.194:39941: BlockingRecv error: recv got EOF from 127.12.222.194:39941 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 02:53:36 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:36 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:36.779193 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:36.773976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.195:43165)
0925 02:53:36.774281 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:36.774293 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:36.774389 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:36.774623 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:36.774625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:36.774798 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:36.774894 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:36.774898 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:36.775575 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:36.775577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:36.776107 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:36.776114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:36.776540 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:36.776543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:36.776605 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:36.777165 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:36.777180 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:36.779080 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":81,"threads_started":1}
W20260925 02:53:36.779317 22500 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:36.774078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:43165 (local address 127.12.222.254:43351)
0925 02:53:36.774210 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:36.774214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:36.774356 (+ 142us) server_negotiation.cc:407] Connection header received
0925 02:53:36.774473 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:36.774475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:36.774517 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:36.774585 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:36.774985 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:36.775485 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:36.776199 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:36.776430 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:36.776521 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:36.779204 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.195:43165: BlockingRecv error: recv got EOF from 127.12.222.195:43165 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 02:53:37 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:37 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:37.448925 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:37.443207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:38693)
0925 02:53:37.443462 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:37.443475 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:37.443567 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:37.443825 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:37.443827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:37.444033 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:37.444129 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:37.444135 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:37.444948 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:37.444951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:37.445489 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:37.445496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:37.445977 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:37.445983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:37.446102 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:37.446748 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:37.446765 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:37.448760 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":82,"threads_started":1}
W20260925 02:53:37.448998 22504 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:37.443264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:38693 (local address 127.12.222.254:43351)
0925 02:53:37.443483 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:37.443487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:37.443536 (+ 49us) server_negotiation.cc:407] Connection header received
0925 02:53:37.443664 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:37.443667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:37.443709 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:37.443786 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:37.444246 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:37.444829 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:37.445594 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:37.445853 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:37.445975 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:37.448870 (+ 2895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.193:38693: BlockingRecv error: recv got EOF from 127.12.222.193:38693 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
Sep 25 02:53:37 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:37 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:37.594863 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:37.589084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:43879)
0925 02:53:37.589430 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:37.589442 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:37.589563 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:37.589838 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:37.589841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:37.590041 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:37.590133 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:37.590138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:37.590914 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:37.590919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:37.591470 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:37.591478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:37.591929 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:37.591933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:37.592016 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:37.592588 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:37.592599 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:37.594755 (+ 2156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":88,"threads_started":1}
W20260925 02:53:37.595000 22504 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:37.589195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:43879 (local address 127.12.222.254:43351)
0925 02:53:37.589359 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:37.589363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:37.589515 (+ 152us) server_negotiation.cc:407] Connection header received
0925 02:53:37.589670 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:37.589674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:37.589722 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:37.589812 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:37.590257 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:37.590804 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:37.591587 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:37.591817 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:37.591935 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:37.594879 (+ 2944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.194:43879: BlockingRecv error: recv got EOF from 127.12.222.194:43879 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 02:53:37 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:37 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:37.786939 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:37.779914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.195:46231)
0925 02:53:37.780207 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:37.780219 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:37.780391 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:37.780650 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:37.780653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:37.780875 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:37.780976 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:37.780981 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:37.782104 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:37.782108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:37.783073 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:37.783087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:37.783785 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:37.783789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:37.783893 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:37.784467 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:37.784481 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:37.786758 (+ 2277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":71,"threads_started":1}
W20260925 02:53:37.787004 22504 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:37.779939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:46231 (local address 127.12.222.254:43351)
0925 02:53:37.780071 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:37.780082 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:37.780317 (+ 235us) server_negotiation.cc:407] Connection header received
0925 02:53:37.780489 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:37.780492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:37.780540 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:37.780627 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:37.781089 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:37.781991 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:37.783249 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:37.783649 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:37.783773 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:37.786889 (+ 3116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.195:46231: BlockingRecv error: recv got EOF from 127.12.222.195:46231 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 02:53:38 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:38 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:38.458667 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:38.449725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:47205)
0925 02:53:38.450002 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:38.450027 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:38.450134 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:38.450360 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:38.450362 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:38.450521 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:38.450609 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:38.450614 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:38.451372 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:38.451374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:38.451866 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:38.451872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:38.452286 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:38.452292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:38.452372 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:38.456767 (+ 4395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:38.456779 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:38.458506 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
W20260925 02:53:38.458745 22508 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:38.449785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:47205 (local address 127.12.222.254:43351)
0925 02:53:38.450042 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:38.450045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:38.450075 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:53:38.450212 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:38.450214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:38.450254 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:38.450325 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:38.450699 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:38.451271 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:38.451959 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:38.452160 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:38.452283 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:38.458624 (+ 6341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.193:47205: BlockingRecv error: recv got EOF from 127.12.222.193:47205 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":61,"threads_started":1}
Sep 25 02:53:38 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:38 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:38.601608 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:38.595879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:33825)
0925 02:53:38.596205 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:38.596221 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:38.596385 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:38.596692 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:38.596695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:38.596911 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:38.597007 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:38.597013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:38.597769 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:38.597774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:38.598328 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:38.598336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:38.598789 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:38.598792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:38.598868 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:38.599465 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:38.599479 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:38.601445 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":133,"threads_started":1}
W20260925 02:53:38.601666 22508 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:38.596051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:33825 (local address 127.12.222.254:43351)
0925 02:53:38.596204 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:38.596208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:38.596324 (+ 116us) server_negotiation.cc:407] Connection header received
0925 02:53:38.596499 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:38.596502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:38.596556 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:38.596666 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:38.597128 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:38.597660 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:38.598449 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:38.598683 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:38.598787 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:38.601560 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.194:33825: BlockingRecv error: recv got EOF from 127.12.222.194:33825 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 02:53:38 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:38 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:38.792779 22510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:38.787834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.195:48255)
0925 02:53:38.788109 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:38.788121 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:38.788218 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:38.788466 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:38.788468 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:38.788640 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:38.788737 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:38.788741 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:38.789432 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:38.789435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:38.789960 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:38.789967 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:38.790348 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:38.790350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:38.790405 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:38.790926 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:38.790936 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:38.792668 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":88,"threads_started":1}
W20260925 02:53:38.792906 22508 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:38.787937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:48255 (local address 127.12.222.254:43351)
0925 02:53:38.788113 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:38.788117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:38.788188 (+ 71us) server_negotiation.cc:407] Connection header received
0925 02:53:38.788313 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:38.788315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:38.788359 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:38.788422 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:38.788830 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:38.789324 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:38.790054 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:38.790262 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:38.790329 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:38.792788 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.195:48255: BlockingRecv error: recv got EOF from 127.12.222.195:48255 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 25 02:53:39 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:39 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:39.464887 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:39.459400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:37411)
0925 02:53:39.459713 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:39.459731 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:39.459844 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:39.460113 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:39.460115 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:39.460329 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:39.460441 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:39.460446 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:39.461155 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:39.461157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:39.461708 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:39.461714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:39.462289 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:39.462293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:39.462386 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:39.462906 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:39.462921 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:39.464778 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":65,"threads_started":1}
W20260925 02:53:39.465040 22512 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:39.459583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:37411 (local address 127.12.222.254:43351)
0925 02:53:39.459908 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:39.459911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:39.459923 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:53:39.459958 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:39.459960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:39.459998 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:39.460069 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:39.460545 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:39.461064 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:39.461859 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:39.462172 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:39.462271 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:39.464908 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.193:37411: BlockingRecv error: recv got EOF from 127.12.222.193:37411 (error 108)
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":73,"threads_started":1}
Sep 25 02:53:39 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:39 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:39.607391 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:39.602468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:53055)
0925 02:53:39.602734 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:39.602745 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:39.602838 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:39.603099 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:39.603101 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:39.603256 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:39.603346 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:39.603351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:39.604033 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:39.604035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:39.604598 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:39.604606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:39.604987 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:39.604989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:39.605047 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:39.605544 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:39.605555 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:39.607285 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":102,"threads_started":1}
W20260925 02:53:39.607555 22512 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:39.602527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:53055 (local address 127.12.222.254:43351)
0925 02:53:39.602665 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:39.602668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:39.602820 (+ 152us) server_negotiation.cc:407] Connection header received
0925 02:53:39.602903 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:39.602905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:39.602964 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:39.603044 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:39.603449 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:39.603912 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:39.604690 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:39.604893 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:39.604972 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:39.607405 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.194:53055: BlockingRecv error: recv got EOF from 127.12.222.194:53055 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 02:53:39 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:39 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.195@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:39.798621 22514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:39.793618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.195:58805)
0925 02:53:39.793879 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:39.793891 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:39.793986 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:39.794200 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:39.794202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:39.794370 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:39.794476 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:39.794481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:39.795191 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:39.795193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:39.795766 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:39.795773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:39.796181 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:39.796184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:39.796266 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:39.796731 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:39.796742 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:39.798510 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":95,"threads_started":1}
W20260925 02:53:39.798753 22512 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:39.793695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:58805 (local address 127.12.222.254:43351)
0925 02:53:39.793822 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:39.793826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:39.793957 (+ 131us) server_negotiation.cc:407] Connection header received
0925 02:53:39.794050 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:39.794052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:39.794096 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:39.794160 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:39.794575 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:39.795102 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:39.795869 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:39.796088 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:39.796162 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:39.798638 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.195:58805: BlockingRecv error: recv got EOF from 127.12.222.195:58805 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 02:53:40 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:40 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.193@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:40.470577 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:40.465747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.193:34361)
0925 02:53:40.466012 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:40.466025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:40.466103 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:40.466325 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:40.466327 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:40.466489 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:40.466579 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:40.466584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:40.467268 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:40.467270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:40.467836 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:40.467843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:40.468262 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:40.468265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:40.468324 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:40.468780 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:40.468790 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:40.470461 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
W20260925 02:53:40.470700 22516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:40.465824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:34361 (local address 127.12.222.254:43351)
0925 02:53:40.466088 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:40.466091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:40.466103 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:53:40.466182 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:40.466184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:40.466224 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:40.466293 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:40.466666 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:40.467181 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:40.467931 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:40.468157 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:40.468282 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:40.470574 (+ 2292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.193:34361: BlockingRecv error: recv got EOF from 127.12.222.193:34361 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":105,"threads_started":1}
Sep 25 02:53:40 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:53:40 dist-test-slave-252r krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.222.194@KRBTEST.COM for oryx/127.12.222.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:53:40.612973 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:40.608315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43351 (local address 127.12.222.194:58569)
0925 02:53:40.608721 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:40.608735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:40.608822 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:40.609001 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:40.609003 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:40.609159 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:40.609244 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:40.609248 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:40.609895 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:40.609897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:40.610377 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:40.610384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:40.610734 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:40.610736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:40.610790 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:40.611226 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:40.611236 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:40.612853 (+ 1617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.222.254:43351: Server oryx/127.12.222.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":106,"threads_started":1}
W20260925 02:53:40.613088 22516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:40.608363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:58569 (local address 127.12.222.254:43351)
0925 02:53:40.608505 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:40.608509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:40.608794 (+ 285us) server_negotiation.cc:407] Connection header received
0925 02:53:40.608882 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:40.608884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:40.608932 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:40.608990 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:40.609329 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:40.609811 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:40.610466 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:40.610651 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:40.610714 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:40.612983 (+ 2269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.222.194:58569: BlockingRecv error: recv got EOF from 127.12.222.194:58569 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:53:40.698016 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 22037
I20260925 02:53:40.703655 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 22191
I20260925 02:53:40.708695 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 22323
I20260925 02:53:40.713562 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 21967
2026-09-25T02:53:40Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17676 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T02:53:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:40Z Disabled control of system clock
I20260925 02:53:40.742516 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:45683
--webserver_interface=127.12.222.254
--webserver_port=0
--builtin_ntp_servers=127.12.222.212:46171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:45683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 02:53:40.844987 22521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:40.845194 22521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:40.845224 22521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:40.848925 22521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:40.848965 22521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:40.848979 22521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:40.848994 22521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:40.849007 22521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:40.853127 22521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:45683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:45683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:40.853942 22521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:40.855098 22521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:40.862788 22526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:40.862789 22530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:40.862809 22527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:40.863168 22521 server_base.cc:1034] running on GCE node
I20260925 02:53:40.863657 22521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:40.864639 22521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:40.865824 22521 hybrid_clock.cc:648] HybridClock initialized: now 1790304820865807 us; error 35 us; skew 500 ppm
I20260925 02:53:40.867522 22521 webserver.cc:460] Webserver started at http://127.12.222.254:32913/ using document root <none> and password file <none>
I20260925 02:53:40.868032 22521 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:40.868126 22521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:40.868383 22521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:40.870138 22521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "dbb520fb17a94a56a6cbf0d40391aa8d"
format_stamp: "Formatted at 2026-09-25 02:53:40 on dist-test-slave-252r"
server_key: "5e4d3dd2e4014a081a94b2b8cd3f9482"
server_key_iv: "02b353d229fe433287e3032a380744e2"
server_key_version: "encryptionkey@0"
I20260925 02:53:40.870664 22521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "dbb520fb17a94a56a6cbf0d40391aa8d"
format_stamp: "Formatted at 2026-09-25 02:53:40 on dist-test-slave-252r"
server_key: "5e4d3dd2e4014a081a94b2b8cd3f9482"
server_key_iv: "02b353d229fe433287e3032a380744e2"
server_key_version: "encryptionkey@0"
I20260925 02:53:40.873901 22521 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:53:40.875989 22536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:40.877118 22521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:40.877250 22521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "dbb520fb17a94a56a6cbf0d40391aa8d"
format_stamp: "Formatted at 2026-09-25 02:53:40 on dist-test-slave-252r"
server_key: "5e4d3dd2e4014a081a94b2b8cd3f9482"
server_key_iv: "02b353d229fe433287e3032a380744e2"
server_key_version: "encryptionkey@0"
I20260925 02:53:40.877379 22521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:40.890302 22521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:40.890839 22521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:40.891010 22521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:40.897197 22521 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:45683
I20260925 02:53:40.897233 22588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:45683 every 8 connection(s)
I20260925 02:53:40.897962 22521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:40.900622 22589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:40.905561 22589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d: Bootstrap starting.
I20260925 02:53:40.907105 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 22521
I20260925 02:53:40.907196 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:53:40.907478 13179 external_mini_cluster.cc:1428] Setting key 746717f8ce2b602230be9892e715bea8
I20260925 02:53:40.907843 22589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:40.908679 22589 log.cc:824] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:40.910589 22589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d: No bootstrap required, opened a new log
I20260925 02:53:40.914345 22589 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb520fb17a94a56a6cbf0d40391aa8d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 45683 } }
I20260925 02:53:40.914558 22589 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:40.914604 22589 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb520fb17a94a56a6cbf0d40391aa8d, State: Initialized, Role: FOLLOWER
I20260925 02:53:40.915017 22589 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb520fb17a94a56a6cbf0d40391aa8d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 45683 } }
I20260925 02:53:40.915164 22589 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:40.915311 22589 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:40.915410 22589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:40.916394 22589 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb520fb17a94a56a6cbf0d40391aa8d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 45683 } }
I20260925 02:53:40.916472 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:40.908478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.12.222.254:45683)
0925 02:53:40.909099 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:40.909113 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:40.909144 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:53:40.909825 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:40.909847 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:40.910198 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:40.910517 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:40.910885 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:40.912208 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:40.913006 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:40.913713 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:40.913865 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:40.914580 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:40.914586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:40.914600 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:40.915012 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:40.915072 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:40.915095 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:40.915186 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:40.915344 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 02:53:40.915585 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
I20260925 02:53:40.916729 22589 leader_election.cc:302] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbb520fb17a94a56a6cbf0d40391aa8d; no voters:
I20260925 02:53:40.917039 22589 leader_election.cc:288] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:53:40.917140 22594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:40.917340 22594 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d [term 1 LEADER]: Becoming Leader. State: Replica: dbb520fb17a94a56a6cbf0d40391aa8d, State: Running, Role: LEADER
I20260925 02:53:40.917681 22594 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb520fb17a94a56a6cbf0d40391aa8d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 45683 } }
I20260925 02:53:40.918210 22589 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:40.918773 22596 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbb520fb17a94a56a6cbf0d40391aa8d. Latest consensus state: current_term: 1 leader_uuid: "dbb520fb17a94a56a6cbf0d40391aa8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb520fb17a94a56a6cbf0d40391aa8d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 45683 } } }
I20260925 02:53:40.918890 22596 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:40.919541 22595 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbb520fb17a94a56a6cbf0d40391aa8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb520fb17a94a56a6cbf0d40391aa8d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 45683 } } }
I20260925 02:53:40.919644 22595 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:40.920257 22600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:40.922511 22600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:40.926998 22600 catalog_manager.cc:1347] Generated new cluster ID: bc4bd6a6e85249e1b8a6698d37765785
I20260925 02:53:40.927103 22600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:40.935729 22600 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:40.936499 22600 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:40.947595 22600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbb520fb17a94a56a6cbf0d40391aa8d: Generated new TSK 0
I20260925 02:53:40.948212 22600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:53:40.958739 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:45683
--builtin_ntp_servers=127.12.222.212:46171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 02:53:41.056090 22613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:41.056366 22613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:41.056439 22613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:41.060582 22613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:41.060647 22613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:41.060717 22613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:53:41.064956 22613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:45683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:41.065793 22613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:41.067090 22613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:41.075568 22619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:41.075596 22618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:41.075632 22613 server_base.cc:1034] running on GCE node
W20260925 02:53:41.075596 22621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:41.076118 22613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:41.076689 22613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:41.077877 22613 hybrid_clock.cc:648] HybridClock initialized: now 1790304821077860 us; error 29 us; skew 500 ppm
I20260925 02:53:41.079506 22613 webserver.cc:460] Webserver started at http://127.12.222.193:46127/ using document root <none> and password file <none>
I20260925 02:53:41.080022 22613 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:41.080080 22613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:41.080367 22613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:41.082351 22613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "734e89e123b346a186e79f6e968914c6"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-252r"
server_key: "057e4cc974f283b57ad550ad5a60f5c6"
server_key_iv: "bfa9dbb8e55c035afd95498908e88412"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.082870 22613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "734e89e123b346a186e79f6e968914c6"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-252r"
server_key: "057e4cc974f283b57ad550ad5a60f5c6"
server_key_iv: "bfa9dbb8e55c035afd95498908e88412"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.086030 22613 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 02:53:41.088210 22627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.089275 22613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:41.089401 22613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "734e89e123b346a186e79f6e968914c6"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-252r"
server_key: "057e4cc974f283b57ad550ad5a60f5c6"
server_key_iv: "bfa9dbb8e55c035afd95498908e88412"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.089500 22613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:41.100646 22613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:41.101190 22613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:41.101369 22613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:41.101856 22613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:41.102676 22613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:41.102739 22613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.102798 22613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:41.102837 22613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.110571 22613 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:37495
I20260925 02:53:41.110597 22741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:37495 every 8 connection(s)
I20260925 02:53:41.111275 22613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:53:41.113991 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 22613
I20260925 02:53:41.114091 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:53:41.114396 13179 external_mini_cluster.cc:1428] Setting key 2f5466e35ed8a99f50ff7a87704adfec
I20260925 02:53:41.116564 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:45683
--builtin_ntp_servers=127.12.222.212:46171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 02:53:41.120781 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:41.112760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:43935 (local address 127.12.222.254:45683)
0925 02:53:41.112888 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:41.112891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:41.113730 (+ 839us) server_negotiation.cc:407] Connection header received
0925 02:53:41.115339 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:41.115343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:41.115390 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:41.115472 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:41.116476 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.117099 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.118662 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.119051 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.119158 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:41.120158 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:41.120165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:41.120171 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:41.120187 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:41.120212 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:41.120247 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:41.120353 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:41.120512 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 02:53:41.120586 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:53:41.121414 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:41.113057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:45683 (local address 127.12.222.193:43935)
0925 02:53:41.113594 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:41.113628 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:41.115152 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:41.115627 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:41.115633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:41.115716 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:41.116334 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:41.116354 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.117225 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.117232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:41.118398 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:41.118457 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.119186 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.119189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:41.119357 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:41.119889 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:53:41.119891 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:41.119922 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:53:41.119976 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:41.120346 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:41.120357 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:41.120363 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:53:41.120477 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 02:53:41.120661 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":94,"threads_started":1}
I20260925 02:53:41.122506 22742 heartbeater.cc:347] Connected to a master server at 127.12.222.254:45683
I20260925 02:53:41.122700 22742 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:41.123128 22742 heartbeater.cc:511] Master 127.12.222.254:45683 requested a full tablet report, sending...
I20260925 02:53:41.124511 22553 ts_manager.cc:194] Registered new tserver with Master: 734e89e123b346a186e79f6e968914c6 (127.12.222.193:37495)
I20260925 02:53:41.125533 22553 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.222.193:43935
W20260925 02:53:41.220374 22745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:41.220583 22745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:41.220641 22745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:41.224130 22745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:41.224187 22745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:41.224293 22745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:53:41.228500 22745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:45683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:41.229300 22745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:41.230446 22745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:41.238806 22751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:41.238835 22750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:41.239075 22745 server_base.cc:1034] running on GCE node
W20260925 02:53:41.238798 22753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:41.239576 22745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:41.240139 22745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:41.241283 22745 hybrid_clock.cc:648] HybridClock initialized: now 1790304821241260 us; error 34 us; skew 500 ppm
I20260925 02:53:41.242738 22745 webserver.cc:460] Webserver started at http://127.12.222.194:45245/ using document root <none> and password file <none>
I20260925 02:53:41.243177 22745 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:41.243239 22745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:41.243366 22745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:41.245071 22745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "9cf334fdea714a938029c35b59b70332"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-252r"
server_key: "0ae1805d717aba3e63895a55444529a9"
server_key_iv: "d3065815dd5bca23712952d83cf2dcbc"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.245512 22745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "9cf334fdea714a938029c35b59b70332"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-252r"
server_key: "0ae1805d717aba3e63895a55444529a9"
server_key_iv: "d3065815dd5bca23712952d83cf2dcbc"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.248608 22745 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:53:41.250576 22759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.251600 22745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:41.251713 22745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "9cf334fdea714a938029c35b59b70332"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-252r"
server_key: "0ae1805d717aba3e63895a55444529a9"
server_key_iv: "d3065815dd5bca23712952d83cf2dcbc"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.251807 22745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:41.281993 22745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:41.282526 22745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:41.282684 22745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:41.283190 22745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:41.283990 22745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:41.284032 22745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.284090 22745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:41.284125 22745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.291917 22745 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:40643
I20260925 02:53:41.291942 22873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:40643 every 8 connection(s)
I20260925 02:53:41.292852 22745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:53:41.300585 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 22745
I20260925 02:53:41.300711 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:53:41.301079 13179 external_mini_cluster.cc:1428] Setting key 20cbaa775b50901449a3707f6e6f0383
I20260925 02:53:41.301775 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:41.294603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:38309 (local address 127.12.222.254:45683)
0925 02:53:41.294741 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:41.294745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:41.295691 (+ 946us) server_negotiation.cc:407] Connection header received
0925 02:53:41.297069 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:41.297072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:41.297134 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:41.297235 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:41.298137 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.298651 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.299718 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.299927 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.300030 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:41.301255 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:41.301258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:41.301260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:41.301273 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:41.301291 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:41.301299 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:41.301360 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:41.301572 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 02:53:41.301634 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:53:41.302641 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:41.294902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:45683 (local address 127.12.222.194:38309)
0925 02:53:41.295540 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:41.295579 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:41.296825 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:41.297407 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:41.297414 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:41.297501 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:41.297972 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:41.297985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.298773 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.298777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:41.299596 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:41.299604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.300070 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.300074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:41.300222 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:41.300947 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:53:41.300950 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:41.301003 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:53:41.301078 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:41.301393 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:41.301403 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:41.301409 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:53:41.301550 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 02:53:41.301805 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":122,"threads_started":1}
I20260925 02:53:41.303184 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:45683
--builtin_ntp_servers=127.12.222.212:46171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 02:53:41.303824 22874 heartbeater.cc:347] Connected to a master server at 127.12.222.254:45683
I20260925 02:53:41.304059 22874 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:41.304678 22874 heartbeater.cc:511] Master 127.12.222.254:45683 requested a full tablet report, sending...
I20260925 02:53:41.305747 22553 ts_manager.cc:194] Registered new tserver with Master: 9cf334fdea714a938029c35b59b70332 (127.12.222.194:40643)
I20260925 02:53:41.306285 22553 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.222.194:38309
W20260925 02:53:41.403774 22877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:41.403957 22877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:41.403986 22877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:41.407501 22877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:41.407542 22877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:41.407589 22877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:53:41.411681 22877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:45683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:41.412498 22877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:41.413589 22877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:41.422359 22885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:41.422343 22882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:41.422360 22883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:41.422475 22877 server_base.cc:1034] running on GCE node
I20260925 02:53:41.422906 22877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:41.423501 22877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:41.424666 22877 hybrid_clock.cc:648] HybridClock initialized: now 1790304821424656 us; error 25 us; skew 500 ppm
I20260925 02:53:41.426117 22877 webserver.cc:460] Webserver started at http://127.12.222.195:41857/ using document root <none> and password file <none>
I20260925 02:53:41.426591 22877 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:41.426674 22877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:41.426867 22877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:41.428680 22877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "794062290776410f8dee350d34612eb6"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-252r"
server_key: "21575ed96a1a628cc7a6bcd90b72a37e"
server_key_iv: "0def2ee26e9e15428b3402ed4996ca00"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.429164 22877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "794062290776410f8dee350d34612eb6"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-252r"
server_key: "21575ed96a1a628cc7a6bcd90b72a37e"
server_key_iv: "0def2ee26e9e15428b3402ed4996ca00"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.432399 22877 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:53:41.434489 22891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.435668 22877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:41.435753 22877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "794062290776410f8dee350d34612eb6"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-252r"
server_key: "21575ed96a1a628cc7a6bcd90b72a37e"
server_key_iv: "0def2ee26e9e15428b3402ed4996ca00"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.435829 22877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:41.455751 22877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:41.456288 22877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:41.456418 22877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:41.456842 22877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:41.457659 22877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:41.457710 22877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.457746 22877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:41.457767 22877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.465752 22877 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:43127
I20260925 02:53:41.465785 23005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:43127 every 8 connection(s)
I20260925 02:53:41.466539 22877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:53:41.469080 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 22877
I20260925 02:53:41.469184 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:53:41.469470 13179 external_mini_cluster.cc:1428] Setting key 0b7d74f3403048a6ed8c96f321588954
I20260925 02:53:41.475390 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:41.468254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:39435 (local address 127.12.222.254:45683)
0925 02:53:41.468382 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:41.468387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:41.469423 (+ 1036us) server_negotiation.cc:407] Connection header received
0925 02:53:41.471029 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:41.471033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:41.471089 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:41.471189 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:41.472023 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.472584 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.473625 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.473843 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.473949 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:41.474938 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:41.474942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:41.474945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:41.474956 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:41.474981 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:41.474989 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:41.475063 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:41.475209 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 02:53:41.475263 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 02:53:41.476091 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:41.468624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:45683 (local address 127.12.222.195:39435)
0925 02:53:41.469289 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:41.469326 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:41.470831 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:41.471345 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:41.471353 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:41.471450 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:41.471884 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:41.471895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.472693 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.472696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:41.473504 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:41.473512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:41.473989 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:41.473993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:41.474140 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:41.474704 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:53:41.474705 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:41.474739 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:53:41.474794 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:41.475065 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:41.475077 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:41.475085 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:53:41.475208 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 02:53:41.475449 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
I20260925 02:53:41.477041 23006 heartbeater.cc:347] Connected to a master server at 127.12.222.254:45683
I20260925 02:53:41.477246 23006 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:41.477672 23006 heartbeater.cc:511] Master 127.12.222.254:45683 requested a full tablet report, sending...
I20260925 02:53:41.478559 22553 ts_manager.cc:194] Registered new tserver with Master: 794062290776410f8dee350d34612eb6 (127.12.222.195:43127)
I20260925 02:53:41.479054 22553 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.222.195:39435
I20260925 02:53:41.482816 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:53:41.485286 22592 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:41.484502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50536 (local address 127.12.222.254:45683)
0925 02:53:41.484641 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:41.484645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:41.484782 (+ 137us) server_negotiation.cc:407] Connection header received
0925 02:53:41.484868 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:41.484870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:41.484902 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:41.484967 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:41.485177 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50536: BlockingRecv error: recv got EOF from 127.0.0.1:50536 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:53:41.485579 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 22613
I20260925 02:53:41.492589 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 22745
I20260925 02:53:41.498284 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 22877
I20260925 02:53:41.502899 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 22521
2026-09-25T02:53:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (787 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T02:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:41Z Disabled control of system clock
I20260925 02:53:41.530134 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:44981
--webserver_interface=127.12.222.254
--webserver_port=0
--builtin_ntp_servers=127.12.222.212:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:44981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 02:53:41.631037 23017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:41.631253 23017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:41.631333 23017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:41.634996 23017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:41.635039 23017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:41.635072 23017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:41.635107 23017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:41.635151 23017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:41.639497 23017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:44981
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:44981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:41.640414 23017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:41.641616 23017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:41.648810 23023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:41.648800 23025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:41.648838 23022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:41.648981 23017 server_base.cc:1034] running on GCE node
I20260925 02:53:41.649550 23017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:41.650467 23017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:41.651648 23017 hybrid_clock.cc:648] HybridClock initialized: now 1790304821651634 us; error 36 us; skew 500 ppm
I20260925 02:53:41.653317 23017 webserver.cc:460] Webserver started at http://127.12.222.254:44301/ using document root <none> and password file <none>
I20260925 02:53:41.653851 23017 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:41.653973 23017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:41.654186 23017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:41.656152 23017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "a764a77315984baf9fca390ad83d9458"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-252r"
server_key: "dfc95830e699523bc8dc61117fbdff8a"
server_key_iv: "528d49a5b08498d826e26b43d45f02fa"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.656740 23017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "a764a77315984baf9fca390ad83d9458"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-252r"
server_key: "dfc95830e699523bc8dc61117fbdff8a"
server_key_iv: "528d49a5b08498d826e26b43d45f02fa"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.659940 23017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:53:41.662077 23031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.663056 23017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 02:53:41.663182 23017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "a764a77315984baf9fca390ad83d9458"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-252r"
server_key: "dfc95830e699523bc8dc61117fbdff8a"
server_key_iv: "528d49a5b08498d826e26b43d45f02fa"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.663281 23017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:41.669329 23017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:41.669977 23017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:41.670181 23017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:41.677341 23017 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:44981
I20260925 02:53:41.677350 23083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:44981 every 8 connection(s)
I20260925 02:53:41.678172 23017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:41.681075 23084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:41.684726 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 23017
I20260925 02:53:41.684835 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:53:41.685128 13179 external_mini_cluster.cc:1428] Setting key f5e3721accb37811e2f64b3b5597d5a0
I20260925 02:53:41.686363 23084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458: Bootstrap starting.
I20260925 02:53:41.688406 23084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:41.689186 23084 log.cc:824] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:41.690251 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:41.686442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.12.222.254:44981)
0925 02:53:41.687011 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:41.687027 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:41.687058 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:53:41.687620 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:41.687635 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:41.687943 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:41.688289 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:41.688972 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:41.688981 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:41.688991 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:41.688998 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:41.689044 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:41.689073 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:41.689178 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:41.689338 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 02:53:41.689569 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":112,"threads_started":1}
I20260925 02:53:41.690903 23084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458: No bootstrap required, opened a new log
I20260925 02:53:41.694259 23084 raft_consensus.cc:348] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a764a77315984baf9fca390ad83d9458" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 44981 } }
I20260925 02:53:41.694478 23084 raft_consensus.cc:374] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:41.694550 23084 raft_consensus.cc:729] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a764a77315984baf9fca390ad83d9458, State: Initialized, Role: FOLLOWER
I20260925 02:53:41.694911 23084 consensus_queue.cc:260] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a764a77315984baf9fca390ad83d9458" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 44981 } }
I20260925 02:53:41.695055 23084 raft_consensus.cc:388] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:41.695132 23084 raft_consensus.cc:482] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:41.695241 23084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:41.695998 23084 raft_consensus.cc:504] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a764a77315984baf9fca390ad83d9458" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 44981 } }
I20260925 02:53:41.696364 23084 leader_election.cc:302] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a764a77315984baf9fca390ad83d9458; no voters:
I20260925 02:53:41.696677 23084 leader_election.cc:288] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:53:41.696786 23089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:41.697034 23089 raft_consensus.cc:686] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458 [term 1 LEADER]: Becoming Leader. State: Replica: a764a77315984baf9fca390ad83d9458, State: Running, Role: LEADER
I20260925 02:53:41.697371 23089 consensus_queue.cc:237] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a764a77315984baf9fca390ad83d9458" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 44981 } }
I20260925 02:53:41.697796 23084 sys_catalog.cc:564] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:41.698810 23091 sys_catalog.cc:455] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a764a77315984baf9fca390ad83d9458. Latest consensus state: current_term: 1 leader_uuid: "a764a77315984baf9fca390ad83d9458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a764a77315984baf9fca390ad83d9458" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 44981 } } }
I20260925 02:53:41.698843 23092 sys_catalog.cc:455] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a764a77315984baf9fca390ad83d9458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a764a77315984baf9fca390ad83d9458" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 44981 } } }
I20260925 02:53:41.698930 23091 sys_catalog.cc:458] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:41.698962 23092 sys_catalog.cc:458] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458 [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:41.699299 23098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:41.702191 23098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:41.707455 23098 catalog_manager.cc:1347] Generated new cluster ID: c46f61dc36c4469782fadc8245b4c22e
I20260925 02:53:41.707507 23098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:41.724977 23098 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:41.725263 23098 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:41.741971 23098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a764a77315984baf9fca390ad83d9458: Generated new TSK 0
I20260925 02:53:41.742584 23098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:53:41.746910 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:44981
--builtin_ntp_servers=127.12.222.212:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 02:53:41.852195 23109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:41.852406 23109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:41.852439 23109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:41.855978 23109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:41.856022 23109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:41.856071 23109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:53:41.861326 23109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:44981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:41.862417 23109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:41.863603 23109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:41.872205 23115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:41.872244 23117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:41.872328 23109 server_base.cc:1034] running on GCE node
W20260925 02:53:41.872263 23114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:41.872828 23109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:41.873350 23109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:41.874545 23109 hybrid_clock.cc:648] HybridClock initialized: now 1790304821874535 us; error 27 us; skew 500 ppm
I20260925 02:53:41.876324 23109 webserver.cc:460] Webserver started at http://127.12.222.193:34275/ using document root <none> and password file <none>
I20260925 02:53:41.876850 23109 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:41.876957 23109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:41.877161 23109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:41.879036 23109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "24ca63ac72484327bd75b56a1b501eb0"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-252r"
server_key: "de3658b4c6062cd673d3415a3924eaf0"
server_key_iv: "404c659dc59b3ce4dc30f1ff516df7b1"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.879591 23109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "24ca63ac72484327bd75b56a1b501eb0"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-252r"
server_key: "de3658b4c6062cd673d3415a3924eaf0"
server_key_iv: "404c659dc59b3ce4dc30f1ff516df7b1"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.882993 23109 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:53:41.885155 23123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.886336 23109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:41.886487 23109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "24ca63ac72484327bd75b56a1b501eb0"
format_stamp: "Formatted at 2026-09-25 02:53:41 on dist-test-slave-252r"
server_key: "de3658b4c6062cd673d3415a3924eaf0"
server_key_iv: "404c659dc59b3ce4dc30f1ff516df7b1"
server_key_version: "encryptionkey@0"
I20260925 02:53:41.886596 23109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:41.890765 23109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:41.891335 23109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:41.891505 23109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:41.892025 23109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:41.892854 23109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:41.892920 23109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.892971 23109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:41.893018 23109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:41.901572 23109 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:35051
I20260925 02:53:41.901598 23237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:35051 every 8 connection(s)
I20260925 02:53:41.902434 23109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:53:41.907979 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:41.904208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:54871 (local address 127.12.222.254:44981)
0925 02:53:41.904409 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:41.904413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:41.905332 (+ 919us) server_negotiation.cc:407] Connection header received
0925 02:53:41.906474 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:41.906477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:41.906512 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:41.906593 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:41.907630 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:41.907633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:41.907635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:41.907644 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:41.907661 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:41.907668 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:41.907743 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:41.907846 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 02:53:41.907881 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260925 02:53:41.908756 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:41.904620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:44981 (local address 127.12.222.193:54871)
0925 02:53:41.905206 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:41.905240 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:41.906274 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:41.906719 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:41.906725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:41.906799 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:41.907397 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:53:41.907398 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:41.907437 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:53:41.907498 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:41.907724 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:41.907727 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:41.907730 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:53:41.907840 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 02:53:41.908010 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":96,"threads_started":1}
I20260925 02:53:41.909921 23238 heartbeater.cc:347] Connected to a master server at 127.12.222.254:44981
I20260925 02:53:41.910131 23238 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:41.910431 23238 heartbeater.cc:511] Master 127.12.222.254:44981 requested a full tablet report, sending...
I20260925 02:53:41.911664 23048 ts_manager.cc:194] Registered new tserver with Master: 24ca63ac72484327bd75b56a1b501eb0 (127.12.222.193:35051)
I20260925 02:53:41.912473 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 23109
I20260925 02:53:41.912575 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:53:41.912897 13179 external_mini_cluster.cc:1428] Setting key f41c729eec2c06fc59f96b70130ec0da
I20260925 02:53:41.914548 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:44981
--builtin_ntp_servers=127.12.222.212:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 02:53:42.013370 23241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:42.013614 23241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:42.013681 23241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:42.017350 23241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:42.017412 23241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:42.017493 23241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:53:42.022023 23241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:44981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:42.022939 23241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:42.024152 23241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:42.032769 23247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:42.032758 23249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:42.032758 23246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:42.032998 23241 server_base.cc:1034] running on GCE node
I20260925 02:53:42.033384 23241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:42.033965 23241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:42.035148 23241 hybrid_clock.cc:648] HybridClock initialized: now 1790304822035133 us; error 26 us; skew 500 ppm
I20260925 02:53:42.036804 23241 webserver.cc:460] Webserver started at http://127.12.222.194:46695/ using document root <none> and password file <none>
I20260925 02:53:42.037315 23241 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:42.037402 23241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:42.037618 23241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:42.039358 23241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "bb5652d65a16441c86aa2fbf55f26ec9"
format_stamp: "Formatted at 2026-09-25 02:53:42 on dist-test-slave-252r"
server_key: "b652ada7ab3aa653f8c088571c8c2b11"
server_key_iv: "494adcdc952d68e2d7e4eb043680693e"
server_key_version: "encryptionkey@0"
I20260925 02:53:42.039860 23241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "bb5652d65a16441c86aa2fbf55f26ec9"
format_stamp: "Formatted at 2026-09-25 02:53:42 on dist-test-slave-252r"
server_key: "b652ada7ab3aa653f8c088571c8c2b11"
server_key_iv: "494adcdc952d68e2d7e4eb043680693e"
server_key_version: "encryptionkey@0"
I20260925 02:53:42.043164 23241 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 02:53:42.045323 23255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:42.046375 23241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:53:42.046509 23241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "bb5652d65a16441c86aa2fbf55f26ec9"
format_stamp: "Formatted at 2026-09-25 02:53:42 on dist-test-slave-252r"
server_key: "b652ada7ab3aa653f8c088571c8c2b11"
server_key_iv: "494adcdc952d68e2d7e4eb043680693e"
server_key_version: "encryptionkey@0"
I20260925 02:53:42.046612 23241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:42.050518 23241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:42.051028 23241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:42.051187 23241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:42.051645 23241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:42.052465 23241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:42.052528 23241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:42.052593 23241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:42.052632 23241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:42.060140 23241 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:35969
I20260925 02:53:42.060164 23369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:35969 every 8 connection(s)
I20260925 02:53:42.060886 23241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:53:42.066272 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.062460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:55645 (local address 127.12.222.254:44981)
0925 02:53:42.062587 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:42.062591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:42.063498 (+ 907us) server_negotiation.cc:407] Connection header received
0925 02:53:42.064698 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:42.064701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:42.064739 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:42.064852 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:42.065841 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:42.065844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:42.065846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:42.065857 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:42.065876 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:42.065890 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:42.065973 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:42.066110 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 02:53:42.066151 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:53:42.066946 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.062783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:44981 (local address 127.12.222.194:55645)
0925 02:53:42.063376 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:42.063408 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:42.064514 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:42.065002 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:42.065008 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:42.065082 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:42.065613 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:53:42.065615 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:42.065648 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:53:42.065704 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:42.065974 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:42.065984 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:42.065991 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:53:42.066147 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 02:53:42.066349 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":114,"threads_started":1}
I20260925 02:53:42.067998 23370 heartbeater.cc:347] Connected to a master server at 127.12.222.254:44981
I20260925 02:53:42.068279 23370 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:42.068652 23370 heartbeater.cc:511] Master 127.12.222.254:44981 requested a full tablet report, sending...
I20260925 02:53:42.069607 23048 ts_manager.cc:194] Registered new tserver with Master: bb5652d65a16441c86aa2fbf55f26ec9 (127.12.222.194:35969)
I20260925 02:53:42.069988 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 23241
I20260925 02:53:42.070099 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:53:42.070370 13179 external_mini_cluster.cc:1428] Setting key 9c78878d81108c79d2eaa27d36a6013b
I20260925 02:53:42.071934 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:44981
--builtin_ntp_servers=127.12.222.212:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 02:53:42.170125 23373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:42.170315 23373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:42.170348 23373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:42.173870 23373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:42.173906 23373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:42.173969 23373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:53:42.178413 23373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:44981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:42.179171 23373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:42.180279 23373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:42.188570 23379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:42.188596 23378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:42.188577 23381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:42.188776 23373 server_base.cc:1034] running on GCE node
I20260925 02:53:42.189373 23373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:42.189908 23373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:42.191143 23373 hybrid_clock.cc:648] HybridClock initialized: now 1790304822191105 us; error 52 us; skew 500 ppm
I20260925 02:53:42.192879 23373 webserver.cc:460] Webserver started at http://127.12.222.195:45825/ using document root <none> and password file <none>
I20260925 02:53:42.193313 23373 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:42.193373 23373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:42.193512 23373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:42.195185 23373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "8d6b32d5404440ab9b48fb2d4dbfaecf"
format_stamp: "Formatted at 2026-09-25 02:53:42 on dist-test-slave-252r"
server_key: "a38ba74fad65d407e7f7507ace558211"
server_key_iv: "bdb184b83a04d6da6a930987df5545cc"
server_key_version: "encryptionkey@0"
I20260925 02:53:42.195633 23373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "8d6b32d5404440ab9b48fb2d4dbfaecf"
format_stamp: "Formatted at 2026-09-25 02:53:42 on dist-test-slave-252r"
server_key: "a38ba74fad65d407e7f7507ace558211"
server_key_iv: "bdb184b83a04d6da6a930987df5545cc"
server_key_version: "encryptionkey@0"
I20260925 02:53:42.199059 23373 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:53:42.201259 23387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:42.202256 23373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:53:42.202379 23373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "8d6b32d5404440ab9b48fb2d4dbfaecf"
format_stamp: "Formatted at 2026-09-25 02:53:42 on dist-test-slave-252r"
server_key: "a38ba74fad65d407e7f7507ace558211"
server_key_iv: "bdb184b83a04d6da6a930987df5545cc"
server_key_version: "encryptionkey@0"
I20260925 02:53:42.202478 23373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:42.206323 23373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:42.206839 23373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:42.207024 23373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:42.207506 23373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:42.208374 23373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:42.208432 23373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:42.208487 23373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:42.208530 23373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:42.216092 23373 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:42667
I20260925 02:53:42.216114 23501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:42667 every 8 connection(s)
I20260925 02:53:42.216805 23373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:53:42.217392 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 23373
I20260925 02:53:42.217514 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:53:42.217799 13179 external_mini_cluster.cc:1428] Setting key 89a18d65874ffe2dcddd7a50e47fa83b
I20260925 02:53:42.222306 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.218682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:37885 (local address 127.12.222.254:44981)
0925 02:53:42.218849 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:42.218853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:42.219887 (+ 1034us) server_negotiation.cc:407] Connection header received
0925 02:53:42.221087 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:42.221089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:42.221121 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:42.221182 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:42.222003 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:42.222006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:42.222007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:53:42.222017 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:42.222033 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:53:42.222040 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:42.222102 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:42.222186 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 02:53:42.222218 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 02:53:42.223021 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:42.219117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:44981 (local address 127.12.222.195:37885)
0925 02:53:42.219753 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:42.219796 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:42.220919 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:42.221290 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:42.221295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:42.221368 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:42.221795 (+ 427us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:53:42.221796 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:42.221828 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:53:42.221883 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:42.222090 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:42.222094 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:42.222097 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:53:42.222200 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 02:53:42.222355 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":121,"threads_started":1}
I20260925 02:53:42.223919 23502 heartbeater.cc:347] Connected to a master server at 127.12.222.254:44981
I20260925 02:53:42.224120 23502 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:42.224479 23502 heartbeater.cc:511] Master 127.12.222.254:44981 requested a full tablet report, sending...
I20260925 02:53:42.225324 23048 ts_manager.cc:194] Registered new tserver with Master: 8d6b32d5404440ab9b48fb2d4dbfaecf (127.12.222.195:42667)
I20260925 02:53:42.231307 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:53:42.233807 23087 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:53:42.233103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.12.222.254:44981)
0925 02:53:42.233200 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:42.233204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:42.233420 (+ 216us) server_negotiation.cc:407] Connection header received
0925 02:53:42.233500 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:42.233503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:42.233528 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:42.233595 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:42.233738 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46970: BlockingRecv error: recv got EOF from 127.0.0.1:46970 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 02:53:42.233997 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 23109
I20260925 02:53:42.239841 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 23241
I20260925 02:53:42.244818 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 23373
I20260925 02:53:42.249786 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 23017
2026-09-25T02:53:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (747 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:53:42 dist-test-slave-252r krb5kdc[23512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:53:42 dist-test-slave-252r krb5kdc[23512](info): set up 2 sockets
Sep 25 02:53:42 dist-test-slave-252r krb5kdc[23512](info): commencing operation
krb5kdc: starting...
W20260925 02:53:43.232932 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:53:43 dist-test-slave-252r krb5kdc[23512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:53:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:53:43.343321 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:42625
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:42625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:43.445430 23525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:43.445667 23525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:43.445741 23525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:43.449648 23525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:53:43.449715 23525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:43.449750 23525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:43.449782 23525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:53:43.449815 23525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:53:43.454378 23525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:42625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:42625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:43.455236 23525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:43.456099 23525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:43.463867 23530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:43.463953 23525 server_base.cc:1034] running on GCE node
W20260925 02:53:43.464035 23531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:43.463867 23533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:43.464701 23525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:43.465623 23525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:43.466799 23525 hybrid_clock.cc:648] HybridClock initialized: now 1790304823466773 us; error 37 us; skew 500 ppm
Sep 25 02:53:43 dist-test-slave-252r krb5kdc[23512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304823, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:43.472663 23525 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:53:43.473387 23525 webserver.cc:460] Webserver started at http://127.12.222.254:45531/ using document root <none> and password file <none>
I20260925 02:53:43.473886 23525 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:43.473973 23525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:43.474189 23525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:43.475967 23525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "9e10658a1341466bbaa8b6529e96cb8b"
format_stamp: "Formatted at 2026-09-25 02:53:43 on dist-test-slave-252r"
server_key: "af417bd598a6084c0bf703b44c657f95"
server_key_iv: "e866f931ad3b446827c71ded522d4647"
server_key_version: "encryptionkey@0"
I20260925 02:53:43.476524 23525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "9e10658a1341466bbaa8b6529e96cb8b"
format_stamp: "Formatted at 2026-09-25 02:53:43 on dist-test-slave-252r"
server_key: "af417bd598a6084c0bf703b44c657f95"
server_key_iv: "e866f931ad3b446827c71ded522d4647"
server_key_version: "encryptionkey@0"
I20260925 02:53:43.479802 23525 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260925 02:53:43.481936 23540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:43.482996 23525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:53:43.483121 23525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "9e10658a1341466bbaa8b6529e96cb8b"
format_stamp: "Formatted at 2026-09-25 02:53:43 on dist-test-slave-252r"
server_key: "af417bd598a6084c0bf703b44c657f95"
server_key_iv: "e866f931ad3b446827c71ded522d4647"
server_key_version: "encryptionkey@0"
I20260925 02:53:43.483217 23525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:43.503486 23525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:43.506407 23525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:43.506601 23525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:43.512552 23525 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:42625
I20260925 02:53:43.512799 23592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:42625 every 8 connection(s)
I20260925 02:53:43.513381 23525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:53:43.515887 23593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:43.518442 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 23525
I20260925 02:53:43.518580 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:53:43.518890 13179 external_mini_cluster.cc:1428] Setting key 856b51ffb28c226621dd299e664f55bf
I20260925 02:53:43.521632 23593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b: Bootstrap starting.
I20260925 02:53:43.523633 23593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:43.524338 23593 log.cc:824] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:43.526046 23593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b: No bootstrap required, opened a new log
Sep 25 02:53:43 dist-test-slave-252r krb5kdc[23512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:43.528875 23593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e10658a1341466bbaa8b6529e96cb8b" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42625 } }
I20260925 02:53:43.529114 23593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:43.529213 23593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e10658a1341466bbaa8b6529e96cb8b, State: Initialized, Role: FOLLOWER
I20260925 02:53:43.529665 23593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e10658a1341466bbaa8b6529e96cb8b" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42625 } }
I20260925 02:53:43.529814 23593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:53:43.529951 23593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:53:43.530076 23593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:43.531092 23593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e10658a1341466bbaa8b6529e96cb8b" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42625 } }
I20260925 02:53:43.531448 23593 leader_election.cc:302] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e10658a1341466bbaa8b6529e96cb8b; no voters:
I20260925 02:53:43.531805 23593 leader_election.cc:288] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:53:43.531865 23598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:43.532111 23598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b [term 1 LEADER]: Becoming Leader. State: Replica: 9e10658a1341466bbaa8b6529e96cb8b, State: Running, Role: LEADER
I20260925 02:53:43.532496 23598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e10658a1341466bbaa8b6529e96cb8b" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42625 } }
I20260925 02:53:43.532768 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:43.520135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.12.222.254:42625)
0925 02:53:43.520556 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:43.520570 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:43.520602 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:53:43.521218 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:43.521233 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:43.521533 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:43.521856 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:43.522644 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:43.523548 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:43.524427 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:43.524866 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:43.525041 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:43.527408 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:43.527413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:43.527426 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:43.527451 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:43.529777 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:43.530309 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:43.530314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:43.530318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:43.530423 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:43.530737 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:43.530740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:43.530741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:43.531094 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:43.531271 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:43.531519 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 02:53:43.531866 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":177,"threads_started":1}
I20260925 02:53:43.533071 23593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:53:43.534056 23600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e10658a1341466bbaa8b6529e96cb8b. Latest consensus state: current_term: 1 leader_uuid: "9e10658a1341466bbaa8b6529e96cb8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e10658a1341466bbaa8b6529e96cb8b" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42625 } } }
I20260925 02:53:43.534168 23600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:43.534499 23599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e10658a1341466bbaa8b6529e96cb8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e10658a1341466bbaa8b6529e96cb8b" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42625 } } }
I20260925 02:53:43.534571 23599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b [sys.catalog]: This master's current role is: LEADER
I20260925 02:53:43.534931 23607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:53:43.537576 23607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:53:43.542479 23607 catalog_manager.cc:1347] Generated new cluster ID: 1816048a797941e0bf2e1d9a874db253
I20260925 02:53:43.542538 23607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:53:43.551890 23607 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:53:43.552992 23607 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:53:43.559334 23607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b: Generated new TSK 0
I20260925 02:53:43.560328 23607 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
I20260925 02:53:43.585498 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:42625
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:43.691541 23619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:43.691758 23619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:43.691797 23619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:43.695729 23619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:43.695777 23619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:43.695871 23619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:53:43.700732 23619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:42625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:43.701661 23619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:43.702560 23619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:43.711402 23628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:43.711390 23625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:43.711423 23619 server_base.cc:1034] running on GCE node
W20260925 02:53:43.711380 23626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:43.711935 23619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:43.712522 23619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:43.713702 23619 hybrid_clock.cc:648] HybridClock initialized: now 1790304823713692 us; error 26 us; skew 500 ppm
Sep 25 02:53:43 dist-test-slave-252r krb5kdc[23512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304823, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:43.716185 23619 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:53:43.716989 23619 webserver.cc:460] Webserver started at http://127.12.222.193:45969/ using document root <none> and password file <none>
I20260925 02:53:43.717535 23619 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:43.717624 23619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:43.717854 23619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:43.719650 23619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "c7c101c28ddb4395a39488ef5d8ae23c"
format_stamp: "Formatted at 2026-09-25 02:53:43 on dist-test-slave-252r"
server_key: "4e6cfc8d816cea2f62ebc13e852b99c5"
server_key_iv: "a56bed19b0bede04e2b43fa30315947c"
server_key_version: "encryptionkey@0"
I20260925 02:53:43.720160 23619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "c7c101c28ddb4395a39488ef5d8ae23c"
format_stamp: "Formatted at 2026-09-25 02:53:43 on dist-test-slave-252r"
server_key: "4e6cfc8d816cea2f62ebc13e852b99c5"
server_key_iv: "a56bed19b0bede04e2b43fa30315947c"
server_key_version: "encryptionkey@0"
I20260925 02:53:43.723423 23619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:53:43.725579 23635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:43.726596 23619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:53:43.726750 23619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "c7c101c28ddb4395a39488ef5d8ae23c"
format_stamp: "Formatted at 2026-09-25 02:53:43 on dist-test-slave-252r"
server_key: "4e6cfc8d816cea2f62ebc13e852b99c5"
server_key_iv: "a56bed19b0bede04e2b43fa30315947c"
server_key_version: "encryptionkey@0"
I20260925 02:53:43.726864 23619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:43.737490 23619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:43.740343 23619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:43.740520 23619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:43.741045 23619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:43.741758 23619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:43.741827 23619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:43.741894 23619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:43.741936 23619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:43.749904 23619 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:37289
I20260925 02:53:43.749953 23748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:37289 every 8 connection(s)
I20260925 02:53:43.750730 23619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:53:43.751175 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 23619
I20260925 02:53:43.751291 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:53:43.751624 13179 external_mini_cluster.cc:1428] Setting key 6446d6a7ab46c00548c1eb14af01b3ef
Sep 25 02:53:43 dist-test-slave-252r krb5kdc[23512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304823, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.222.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.194@KRBTEST.COM" created.
I20260925 02:53:43.765367 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:43.752474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:59359 (local address 127.12.222.254:42625)
0925 02:53:43.752601 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:43.752606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:43.753465 (+ 859us) server_negotiation.cc:407] Connection header received
0925 02:53:43.754733 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:43.754736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:43.754786 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:43.754858 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:43.755949 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:43.756868 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:43.758376 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:43.758782 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:43.758904 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:43.761085 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:43.761089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:43.761092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:43.761119 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:43.762720 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:43.763422 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:43.763426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:43.763428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:43.763531 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:43.764004 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:43.764007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:43.764008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:43.764223 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:43.764318 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:43.764976 (+ 658us) server_negotiation.cc:299] Negotiation successful
0925 02:53:43.765159 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:53:43.766146 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:43.752806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42625 (local address 127.12.222.193:59359)
0925 02:53:43.753324 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:43.753371 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:43.754547 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:43.755009 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:43.755015 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:43.755335 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:43.755808 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:43.755821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:43.757037 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:43.757040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:43.758222 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:43.758233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:43.758925 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:43.758927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:43.759055 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:43.759556 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:43.759570 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:43.760923 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:43.762834 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:43.762839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:43.762850 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:43.763237 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:43.763612 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:43.763615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:43.763617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:43.763870 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:43.764402 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:43.764409 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:43.764682 (+ 273us) client_negotiation.cc:769] Sending connection context
0925 02:53:43.764880 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 02:53:43.765228 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":145,"threads_started":1}
I20260925 02:53:43.767177 23749 heartbeater.cc:347] Connected to a master server at 127.12.222.254:42625
I20260925 02:53:43.767391 23749 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:43.767933 23749 heartbeater.cc:511] Master 127.12.222.254:42625 requested a full tablet report, sending...
I20260925 02:53:43.769584 23557 ts_manager.cc:194] Registered new tserver with Master: c7c101c28ddb4395a39488ef5d8ae23c (127.12.222.193:37289)
I20260925 02:53:43.770602 23557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:59359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
I20260925 02:53:43.779449 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:42625
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:43.882193 23754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:43.882426 23754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:43.882495 23754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:43.886353 23754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:43.886420 23754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:43.886503 23754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:53:43.891366 23754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:42625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:43.892307 23754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:43.893183 23754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:43.901741 23760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:43.901754 23759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:43.901887 23754 server_base.cc:1034] running on GCE node
W20260925 02:53:43.902060 23762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:43.902459 23754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:43.903007 23754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:43.904189 23754 hybrid_clock.cc:648] HybridClock initialized: now 1790304823904175 us; error 28 us; skew 500 ppm
Sep 25 02:53:43 dist-test-slave-252r krb5kdc[23512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304823, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:43.906633 23754 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:53:43.907349 23754 webserver.cc:460] Webserver started at http://127.12.222.194:34639/ using document root <none> and password file <none>
I20260925 02:53:43.907853 23754 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:43.907938 23754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:43.908147 23754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:43.910151 23754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "4c410d5b60fa4646942b8aec94b75545"
format_stamp: "Formatted at 2026-09-25 02:53:43 on dist-test-slave-252r"
server_key: "11efcc04a91fb4e0b0d1aa20f0d232bf"
server_key_iv: "c35f1cf7691fc00aaf82488f692e8694"
server_key_version: "encryptionkey@0"
I20260925 02:53:43.910656 23754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "4c410d5b60fa4646942b8aec94b75545"
format_stamp: "Formatted at 2026-09-25 02:53:43 on dist-test-slave-252r"
server_key: "11efcc04a91fb4e0b0d1aa20f0d232bf"
server_key_iv: "c35f1cf7691fc00aaf82488f692e8694"
server_key_version: "encryptionkey@0"
I20260925 02:53:43.913944 23754 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:53:43.916002 23769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:43.917263 23754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:43.917351 23754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "4c410d5b60fa4646942b8aec94b75545"
format_stamp: "Formatted at 2026-09-25 02:53:43 on dist-test-slave-252r"
server_key: "11efcc04a91fb4e0b0d1aa20f0d232bf"
server_key_iv: "c35f1cf7691fc00aaf82488f692e8694"
server_key_version: "encryptionkey@0"
I20260925 02:53:43.917428 23754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:43.935760 23754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:43.938192 23754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:43.938341 23754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:43.938807 23754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:43.939504 23754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:43.939549 23754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:43.939597 23754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:43.939617 23754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:43.947753 23754 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:36321
I20260925 02:53:43.947801 23883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:36321 every 8 connection(s)
I20260925 02:53:43.948561 23754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:53:43.954643 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 23754
I20260925 02:53:43.954756 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:53:43.955052 13179 external_mini_cluster.cc:1428] Setting key 3bc5e62e83359eca9afb800adaf81895
Sep 25 02:53:43 dist-test-slave-252r krb5kdc[23512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304823, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:43.962287 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:43.950261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:52735 (local address 127.12.222.254:42625)
0925 02:53:43.950391 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:43.950395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:43.951172 (+ 777us) server_negotiation.cc:407] Connection header received
0925 02:53:43.952476 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:43.952479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:43.952534 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:43.952606 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:43.953608 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:43.954116 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:43.955579 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:43.955839 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:43.955945 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:43.958673 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:43.958675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:43.958677 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:43.958696 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:43.960065 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:43.960700 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:43.960717 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:43.960720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:43.960822 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:43.961234 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:43.961236 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:43.961238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:43.961393 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:43.961471 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:43.962096 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 02:53:43.962179 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:53:43.963361 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:43.950602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42625 (local address 127.12.222.194:52735)
0925 02:53:43.951028 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:43.951069 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:43.952282 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:43.952770 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:43.952776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:43.953080 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:43.953475 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:43.953487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:43.954238 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:43.954244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:43.955456 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:43.955465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:43.956003 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:43.956007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:43.956171 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:43.956740 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:43.956760 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:43.958500 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:43.960197 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:43.960205 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:43.960220 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:43.960609 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:43.960928 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:43.960934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:43.960937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:43.961135 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:43.961500 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:43.961509 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:43.961829 (+ 320us) client_negotiation.cc:769] Sending connection context
0925 02:53:43.962055 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 02:53:43.962384 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":108,"threads_started":1}
I20260925 02:53:43.964516 23884 heartbeater.cc:347] Connected to a master server at 127.12.222.254:42625
I20260925 02:53:43.964715 23884 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:43.965113 23884 heartbeater.cc:511] Master 127.12.222.254:42625 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.222.195@KRBTEST.COM; defaulting to no policy
I20260925 02:53:43.966032 23557 ts_manager.cc:194] Registered new tserver with Master: 4c410d5b60fa4646942b8aec94b75545 (127.12.222.194:36321)
I20260925 02:53:43.966542 23557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:52735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
I20260925 02:53:43.982590 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:42625
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:53:44.083602 23889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:53:44.083837 23889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:53:44.083923 23889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:53:44.087662 23889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:53:44.087728 23889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:53:44.087808 23889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:53:44.092396 23889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:42625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:53:44.093261 23889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:53:44.094205 23889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:53:44.103140 23897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:44.103188 23894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:53:44.103152 23895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:53:44.103477 23889 server_base.cc:1034] running on GCE node
I20260925 02:53:44.103761 23889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:53:44.104313 23889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:53:44.105475 23889 hybrid_clock.cc:648] HybridClock initialized: now 1790304824105457 us; error 26 us; skew 500 ppm
Sep 25 02:53:44 dist-test-slave-252r krb5kdc[23512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304824, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:53:44.107966 23889 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:53:44.108917 23889 webserver.cc:460] Webserver started at http://127.12.222.195:39559/ using document root <none> and password file <none>
I20260925 02:53:44.109426 23889 fs_manager.cc:359] Metadata directory not provided
I20260925 02:53:44.109496 23889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:53:44.109732 23889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:53:44.111436 23889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "0be29894cca24c87b3d0f6f41bc6784b"
format_stamp: "Formatted at 2026-09-25 02:53:44 on dist-test-slave-252r"
server_key: "8fc04de6e9928930cf92f54562a17c66"
server_key_iv: "b1ebd04ea9c6e9ae1c7460148cea513a"
server_key_version: "encryptionkey@0"
I20260925 02:53:44.111930 23889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "0be29894cca24c87b3d0f6f41bc6784b"
format_stamp: "Formatted at 2026-09-25 02:53:44 on dist-test-slave-252r"
server_key: "8fc04de6e9928930cf92f54562a17c66"
server_key_iv: "b1ebd04ea9c6e9ae1c7460148cea513a"
server_key_version: "encryptionkey@0"
I20260925 02:53:44.114959 23889 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:53:44.116978 23904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:44.118110 23889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:53:44.118239 23889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "0be29894cca24c87b3d0f6f41bc6784b"
format_stamp: "Formatted at 2026-09-25 02:53:44 on dist-test-slave-252r"
server_key: "8fc04de6e9928930cf92f54562a17c66"
server_key_iv: "b1ebd04ea9c6e9ae1c7460148cea513a"
server_key_version: "encryptionkey@0"
I20260925 02:53:44.118358 23889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:53:44.130450 23889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:53:44.132877 23889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:53:44.133042 23889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:53:44.133572 23889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:53:44.134394 23889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:53:44.134457 23889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:44.134516 23889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:53:44.134562 23889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:53:44.142611 23889 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:36743
I20260925 02:53:44.142637 24018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:36743 every 8 connection(s)
I20260925 02:53:44.143326 23889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:53:44.147750 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 23889
I20260925 02:53:44.147868 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:53:44.148160 13179 external_mini_cluster.cc:1428] Setting key a5ea67ccc3b8a31ae5b8df6f488b564c
Sep 25 02:53:44 dist-test-slave-252r krb5kdc[23512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304824, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:53:44.155304 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.144959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:36695 (local address 127.12.222.254:42625)
0925 02:53:44.145109 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:44.145113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:44.145949 (+ 836us) server_negotiation.cc:407] Connection header received
0925 02:53:44.146642 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:44.146644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:44.146686 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:44.146752 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:44.147765 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.148521 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.149778 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.150036 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.150144 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:44.152313 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:44.152318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:44.152324 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:44.152346 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:44.153534 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:44.153988 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:44.153991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:44.153993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:44.154068 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:44.154367 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:44.154370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:44.154372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:44.154536 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:44.154636 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:44.155049 (+ 413us) server_negotiation.cc:299] Negotiation successful
0925 02:53:44.155153 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:53:44.155928 24021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.145268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42625 (local address 127.12.222.195:36695)
0925 02:53:44.145817 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:44.145850 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:44.146489 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:44.146895 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:44.146901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:44.147215 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:53:44.147628 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:44.147640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.148678 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.148681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:44.149671 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:44.149680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.150181 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.150185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:44.150329 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:44.150815 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:53:44.150829 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:53:44.152141 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:53:44.153642 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:44.153648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:44.153659 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:44.153889 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:44.154153 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:53:44.154156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:53:44.154157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:53:44.154277 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:53:44.154637 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:53:44.154643 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:53:44.154839 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 02:53:44.155007 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 02:53:44.155266 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":100,"threads_started":1}
I20260925 02:53:44.156932 24019 heartbeater.cc:347] Connected to a master server at 127.12.222.254:42625
I20260925 02:53:44.157132 24019 heartbeater.cc:464] Registering TS with master...
I20260925 02:53:44.157567 24019 heartbeater.cc:511] Master 127.12.222.254:42625 requested a full tablet report, sending...
I20260925 02:53:44.158501 23557 ts_manager.cc:194] Registered new tserver with Master: 0be29894cca24c87b3d0f6f41bc6784b (127.12.222.195:36743)
I20260925 02:53:44.158962 23557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:36695
I20260925 02:53:44.161716 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:53:44.169350 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.163350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46942 (local address 127.12.222.254:42625)
0925 02:53:44.163469 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:44.163473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:44.163601 (+ 128us) server_negotiation.cc:407] Connection header received
0925 02:53:44.163681 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:44.163683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:44.163715 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:44.163772 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:53:44.164392 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.164804 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.165481 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.165643 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.165703 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:44.166686 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:53:44.166688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:53:44.166689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:53:44.166706 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:53:44.167810 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:44.168156 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:44.168158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:44.168159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:44.168224 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:53:44.168518 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:53:44.168520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:53:44.168521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:53:44.168727 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:53:44.168833 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:44.169080 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 02:53:44.169161 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:53:44.171599 23557 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:53:44.174326 23557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:53:44.187110 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.181547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:36743 (local address 127.0.0.1:59228)
0925 02:53:44.182241 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:44.182282 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:44.182408 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:44.182925 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:44.182930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:44.182955 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:44.183089 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:44.183096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.184269 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.184272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:44.185515 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:44.185526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.186509 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.186513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:44.186621 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:44.186657 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 02:53:44.186762 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 02:53:44.186925 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":23,"thread_start_us":73,"threads_started":1}
I20260925 02:53:44.187142 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.181649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:36321 (local address 127.0.0.1:40324)
0925 02:53:44.182629 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:44.182642 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:44.182728 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:44.183144 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:44.183146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:44.183165 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:44.183235 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:44.183240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.184512 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.184515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:44.185611 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:44.185620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.186706 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.186710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:44.186795 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:44.186842 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 02:53:44.186918 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 02:53:44.186985 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":40,"thread_start_us":88,"threads_started":1}
I20260925 02:53:44.187160 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.181943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.12.222.194:36321)
0925 02:53:44.182313 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:44.182326 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:44.182732 (+ 406us) server_negotiation.cc:407] Connection header received
0925 02:53:44.182818 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:44.182824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:44.183017 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:44.183172 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:44.183306 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.184377 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.185803 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.186573 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.186771 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:44.186896 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:44.186944 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 02:53:44.187027 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":85,"threads_started":1}
I20260925 02:53:44.187461 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.181916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:37289 (local address 127.0.0.1:60912)
0925 02:53:44.182894 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:44.182906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:44.182971 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:44.183629 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:44.183631 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:44.183648 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:44.183719 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:44.183723 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.184884 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.184887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:44.185944 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:44.185951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.187006 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.187011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:44.187108 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:44.187139 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 02:53:44.187208 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 02:53:44.187307 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":127,"thread_start_us":83,"threads_started":1}
I20260925 02:53:44.187999 24034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.182126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59228 (local address 127.12.222.195:36743)
0925 02:53:44.182454 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:44.182467 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:44.182489 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:53:44.182552 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:44.182556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:44.182735 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:44.183230 (+ 495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:44.183368 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.184124 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.185624 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.186388 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.186509 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:44.186627 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:44.187770 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0925 02:53:44.187857 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":77,"threads_started":1}
I20260925 02:53:44.188010 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.182297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60912 (local address 127.12.222.193:37289)
0925 02:53:44.183210 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:44.183222 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:44.183242 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:53:44.183308 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:44.183312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:44.183488 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:44.183628 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:44.183836 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.184768 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.186053 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.186883 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.187621 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:44.187746 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:44.187790 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 02:53:44.187860 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":119,"threads_started":1}
I20260925 02:53:44.190310 23683 tablet_service.cc:1467] Processing CreateTablet for tablet c4a8778603da420abefc04652825ff8b (DEFAULT_TABLE table=test-table [id=85d13cfd5d2645718d860aaa53754a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:44.190407 23952 tablet_service.cc:1467] Processing CreateTablet for tablet c4a8778603da420abefc04652825ff8b (DEFAULT_TABLE table=test-table [id=85d13cfd5d2645718d860aaa53754a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:44.190414 23818 tablet_service.cc:1467] Processing CreateTablet for tablet c4a8778603da420abefc04652825ff8b (DEFAULT_TABLE table=test-table [id=85d13cfd5d2645718d860aaa53754a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:53:44.191183 23683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4a8778603da420abefc04652825ff8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:44.191289 23952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4a8778603da420abefc04652825ff8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:44.191283 23818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4a8778603da420abefc04652825ff8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:53:44.196476 24036 tablet_bootstrap.cc:492] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c: Bootstrap starting.
I20260925 02:53:44.197553 24037 tablet_bootstrap.cc:492] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b: Bootstrap starting.
I20260925 02:53:44.198364 24038 tablet_bootstrap.cc:492] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545: Bootstrap starting.
I20260925 02:53:44.199461 24037 tablet_bootstrap.cc:654] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:44.199882 24036 tablet_bootstrap.cc:654] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:44.200366 24038 tablet_bootstrap.cc:654] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545: Neither blocks nor log segments found. Creating new log.
I20260925 02:53:44.200649 24037 log.cc:824] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:44.200649 24036 log.cc:824] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:44.201116 24038 log.cc:824] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545: Log is configured to *not* fsync() on all Append() calls
I20260925 02:53:44.202363 24036 tablet_bootstrap.cc:492] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c: No bootstrap required, opened a new log
I20260925 02:53:44.202592 24036 ts_tablet_manager.cc:1397] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20260925 02:53:44.202737 24038 tablet_bootstrap.cc:492] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545: No bootstrap required, opened a new log
I20260925 02:53:44.202940 24038 ts_tablet_manager.cc:1397] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 02:53:44.203615 24037 tablet_bootstrap.cc:492] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b: No bootstrap required, opened a new log
I20260925 02:53:44.203780 24037 ts_tablet_manager.cc:1397] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:53:44.205580 24038 raft_consensus.cc:348] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be29894cca24c87b3d0f6f41bc6784b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36743 } } peers { permanent_uuid: "4c410d5b60fa4646942b8aec94b75545" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36321 } } peers { permanent_uuid: "c7c101c28ddb4395a39488ef5d8ae23c" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 37289 } }
I20260925 02:53:44.205785 24038 raft_consensus.cc:374] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:44.205842 24038 raft_consensus.cc:729] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c410d5b60fa4646942b8aec94b75545, State: Initialized, Role: FOLLOWER
I20260925 02:53:44.206264 24038 consensus_queue.cc:260] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be29894cca24c87b3d0f6f41bc6784b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36743 } } peers { permanent_uuid: "4c410d5b60fa4646942b8aec94b75545" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36321 } } peers { permanent_uuid: "c7c101c28ddb4395a39488ef5d8ae23c" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 37289 } }
I20260925 02:53:44.206251 24036 raft_consensus.cc:348] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be29894cca24c87b3d0f6f41bc6784b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36743 } } peers { permanent_uuid: "4c410d5b60fa4646942b8aec94b75545" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36321 } } peers { permanent_uuid: "c7c101c28ddb4395a39488ef5d8ae23c" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 37289 } }
I20260925 02:53:44.206473 24036 raft_consensus.cc:374] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:44.206532 24036 raft_consensus.cc:729] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c101c28ddb4395a39488ef5d8ae23c, State: Initialized, Role: FOLLOWER
I20260925 02:53:44.206863 23884 heartbeater.cc:503] Master 127.12.222.254:42625 was elected leader, sending a full tablet report...
I20260925 02:53:44.206915 24036 consensus_queue.cc:260] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be29894cca24c87b3d0f6f41bc6784b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36743 } } peers { permanent_uuid: "4c410d5b60fa4646942b8aec94b75545" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36321 } } peers { permanent_uuid: "c7c101c28ddb4395a39488ef5d8ae23c" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 37289 } }
I20260925 02:53:44.207046 24037 raft_consensus.cc:348] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be29894cca24c87b3d0f6f41bc6784b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36743 } } peers { permanent_uuid: "4c410d5b60fa4646942b8aec94b75545" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36321 } } peers { permanent_uuid: "c7c101c28ddb4395a39488ef5d8ae23c" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 37289 } }
I20260925 02:53:44.207278 24037 raft_consensus.cc:374] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:53:44.207333 24037 raft_consensus.cc:729] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be29894cca24c87b3d0f6f41bc6784b, State: Initialized, Role: FOLLOWER
I20260925 02:53:44.207643 23749 heartbeater.cc:503] Master 127.12.222.254:42625 was elected leader, sending a full tablet report...
I20260925 02:53:44.207829 24036 ts_tablet_manager.cc:1428] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260925 02:53:44.208194 24038 ts_tablet_manager.cc:1428] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260925 02:53:44.208518 24037 consensus_queue.cc:260] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be29894cca24c87b3d0f6f41bc6784b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36743 } } peers { permanent_uuid: "4c410d5b60fa4646942b8aec94b75545" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36321 } } peers { permanent_uuid: "c7c101c28ddb4395a39488ef5d8ae23c" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 37289 } }
I20260925 02:53:44.209264 24019 heartbeater.cc:503] Master 127.12.222.254:42625 was elected leader, sending a full tablet report...
I20260925 02:53:44.209477 24037 ts_tablet_manager.cc:1428] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260925 02:53:44.251860 23750 tablet.cc:2369] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:53:44.394215 24020 tablet.cc:2369] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:53:44.449709 23885 tablet.cc:2369] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:53:44.465651 24043 raft_consensus.cc:482] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:53:44.465871 24043 raft_consensus.cc:504] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be29894cca24c87b3d0f6f41bc6784b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36743 } } peers { permanent_uuid: "4c410d5b60fa4646942b8aec94b75545" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36321 } } peers { permanent_uuid: "c7c101c28ddb4395a39488ef5d8ae23c" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 37289 } }
I20260925 02:53:44.467069 24043 leader_election.cc:288] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0be29894cca24c87b3d0f6f41bc6784b (127.12.222.195:36743), 4c410d5b60fa4646942b8aec94b75545 (127.12.222.194:36321)
I20260925 02:53:44.470818 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.467469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:59115 (local address 127.12.222.194:36321)
0925 02:53:44.467584 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:44.467589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:44.467715 (+ 126us) server_negotiation.cc:407] Connection header received
0925 02:53:44.467911 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:44.467915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:44.467968 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:44.468024 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:44.468414 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.468892 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.469954 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.470304 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.470389 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:44.470456 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:44.470638 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 02:53:44.470708 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:53:44.470822 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.467291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:36321 (local address 127.12.222.193:59115)
0925 02:53:44.467633 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:44.467645 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:44.467765 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:44.468067 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:44.468070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:44.468085 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:44.468274 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:44.468281 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.468997 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.469001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:44.469845 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:44.469853 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.470423 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.470428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:44.470515 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:44.470541 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:53:44.470611 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 02:53:44.470660 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":68,"spinlock_wait_cycles":4992,"thread_start_us":89,"threads_started":1}
I20260925 02:53:44.470942 24034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.467424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:36335 (local address 127.12.222.195:36743)
0925 02:53:44.467553 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:44.467557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:44.467792 (+ 235us) server_negotiation.cc:407] Connection header received
0925 02:53:44.467933 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:44.467935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:44.467978 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:44.468064 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:53:44.468450 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.468894 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.469986 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.470366 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.470480 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:44.470524 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:44.470758 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 02:53:44.470823 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:53:44.471304 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.467291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:36743 (local address 127.12.222.193:36335)
0925 02:53:44.467700 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:44.467712 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:53:44.467804 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:53:44.468091 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:53:44.468094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:53:44.468131 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:53:44.468313 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:44.468321 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.468998 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.469001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:44.469851 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:53:44.469860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.470493 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.470496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:53:44.470635 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:44.470657 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:53:44.470725 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 02:53:44.470784 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":100,"spinlock_wait_cycles":768,"thread_start_us":94,"threads_started":1}
I20260925 02:53:44.471495 23838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a8778603da420abefc04652825ff8b" candidate_uuid: "c7c101c28ddb4395a39488ef5d8ae23c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c410d5b60fa4646942b8aec94b75545" is_pre_election: true
I20260925 02:53:44.471808 23838 raft_consensus.cc:2454] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7c101c28ddb4395a39488ef5d8ae23c in term 0.
I20260925 02:53:44.471999 23973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a8778603da420abefc04652825ff8b" candidate_uuid: "c7c101c28ddb4395a39488ef5d8ae23c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be29894cca24c87b3d0f6f41bc6784b" is_pre_election: true
I20260925 02:53:44.472293 23973 raft_consensus.cc:2454] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7c101c28ddb4395a39488ef5d8ae23c in term 0.
I20260925 02:53:44.472290 23637 leader_election.cc:302] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c410d5b60fa4646942b8aec94b75545, c7c101c28ddb4395a39488ef5d8ae23c; no voters:
I20260925 02:53:44.472575 24043 raft_consensus.cc:2781] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:53:44.472667 24043 raft_consensus.cc:482] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:53:44.472718 24043 raft_consensus.cc:3037] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:44.473694 24043 raft_consensus.cc:504] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be29894cca24c87b3d0f6f41bc6784b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36743 } } peers { permanent_uuid: "4c410d5b60fa4646942b8aec94b75545" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36321 } } peers { permanent_uuid: "c7c101c28ddb4395a39488ef5d8ae23c" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 37289 } }
I20260925 02:53:44.474000 24043 leader_election.cc:288] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [CANDIDATE]: Term 1 election: Requested vote from peers 0be29894cca24c87b3d0f6f41bc6784b (127.12.222.195:36743), 4c410d5b60fa4646942b8aec94b75545 (127.12.222.194:36321)
I20260925 02:53:44.474278 23973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a8778603da420abefc04652825ff8b" candidate_uuid: "c7c101c28ddb4395a39488ef5d8ae23c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be29894cca24c87b3d0f6f41bc6784b"
I20260925 02:53:44.474411 23973 raft_consensus.cc:3037] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:44.474433 23838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a8778603da420abefc04652825ff8b" candidate_uuid: "c7c101c28ddb4395a39488ef5d8ae23c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c410d5b60fa4646942b8aec94b75545"
I20260925 02:53:44.474557 23838 raft_consensus.cc:3037] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:53:44.475425 23973 raft_consensus.cc:2454] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7c101c28ddb4395a39488ef5d8ae23c in term 1.
I20260925 02:53:44.475498 23838 raft_consensus.cc:2454] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7c101c28ddb4395a39488ef5d8ae23c in term 1.
I20260925 02:53:44.475739 23639 leader_election.cc:302] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be29894cca24c87b3d0f6f41bc6784b, c7c101c28ddb4395a39488ef5d8ae23c; no voters:
I20260925 02:53:44.475925 24043 raft_consensus.cc:2781] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:53:44.476114 24043 raft_consensus.cc:686] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [term 1 LEADER]: Becoming Leader. State: Replica: c7c101c28ddb4395a39488ef5d8ae23c, State: Running, Role: LEADER
I20260925 02:53:44.476425 24043 consensus_queue.cc:237] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be29894cca24c87b3d0f6f41bc6784b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36743 } } peers { permanent_uuid: "4c410d5b60fa4646942b8aec94b75545" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36321 } } peers { permanent_uuid: "c7c101c28ddb4395a39488ef5d8ae23c" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 37289 } }
I20260925 02:53:44.479168 23556 catalog_manager.cc:5495] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c reported cstate change: term changed from 0 to 1, leader changed from <none> to c7c101c28ddb4395a39488ef5d8ae23c (127.12.222.193). New cstate: current_term: 1 leader_uuid: "c7c101c28ddb4395a39488ef5d8ae23c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be29894cca24c87b3d0f6f41bc6784b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c410d5b60fa4646942b8aec94b75545" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7c101c28ddb4395a39488ef5d8ae23c" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 37289 } health_report { overall_health: HEALTHY } } }
I20260925 02:53:44.544512 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.541465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60916 (local address 127.12.222.193:37289)
0925 02:53:44.541600 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 02:53:44.541604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:53:44.541616 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:53:44.541682 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:53:44.541685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:53:44.541736 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:53:44.541819 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:53:44.542049 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.542524 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.543294 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:53:44.543498 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:53:44.543564 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:53:44.543816 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:53:44.544156 (+ 340us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:53:44.544299 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 02:53:44.544326 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 02:53:44.544368 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:53:44.549698 23973 raft_consensus.cc:1261] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b [term 1 FOLLOWER]: Refusing update from remote peer c7c101c28ddb4395a39488ef5d8ae23c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:44.549708 23838 raft_consensus.cc:1261] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545 [term 1 FOLLOWER]: Refusing update from remote peer c7c101c28ddb4395a39488ef5d8ae23c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:53:44.550349 24043 consensus_queue.cc:1035] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c410d5b60fa4646942b8aec94b75545" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:44.550606 24047 consensus_queue.cc:1035] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0be29894cca24c87b3d0f6f41bc6784b" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:53:44.558048 24052 mvcc.cc:205] Tried to move back new op lower bound from 7333088561349140480 to 7333088561057525760. Current Snapshot: MvccSnapshot[applied={T|T < 7333088561349140480}]
I20260925 02:53:44.564309 23557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:44.564459 23557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:53:44.566529 23557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e10658a1341466bbaa8b6529e96cb8b: Sending DeleteTablet for 3 replicas of tablet c4a8778603da420abefc04652825ff8b
I20260925 02:53:44.567121 23952 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a8778603da420abefc04652825ff8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:44 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:59228
I20260925 02:53:44.567202 23818 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a8778603da420abefc04652825ff8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:44 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:40324
I20260925 02:53:44.567332 23683 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a8778603da420abefc04652825ff8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:53:44 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:60912
I20260925 02:53:44.567608 24060 tablet_replica.cc:330] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b: stopping tablet replica
I20260925 02:53:44.567787 24062 tablet_replica.cc:330] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c: stopping tablet replica
I20260925 02:53:44.567926 24060 raft_consensus.cc:2229] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:44.568065 24062 raft_consensus.cc:2229] T c4a8778603da420abefc04652825ff8b P c7c101c28ddb4395a39488ef5d8ae23c [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:53:44.568284 24060 raft_consensus.cc:2258] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:44.568463 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 23619
I20260925 02:53:44.569258 24061 tablet_replica.cc:330] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545: stopping tablet replica
I20260925 02:53:44.569502 24061 raft_consensus.cc:2229] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:53:44.569802 24061 raft_consensus.cc:2258] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:53:44.570770 24060 ts_tablet_manager.cc:1905] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:44.571575 24061 ts_tablet_manager.cc:1905] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:53:44.573557 24060 ts_tablet_manager.cc:1918] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:44.573647 24060 log.cc:1196] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/c4a8778603da420abefc04652825ff8b
I20260925 02:53:44.574018 24060 ts_tablet_manager.cc:1939] T c4a8778603da420abefc04652825ff8b P 0be29894cca24c87b3d0f6f41bc6784b: Deleting consensus metadata
I20260925 02:53:44.574286 24061 ts_tablet_manager.cc:1918] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:53:44.574378 24061 log.cc:1196] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/c4a8778603da420abefc04652825ff8b
I20260925 02:53:44.574676 24061 ts_tablet_manager.cc:1939] T c4a8778603da420abefc04652825ff8b P 4c410d5b60fa4646942b8aec94b75545: Deleting consensus metadata
I20260925 02:53:44.575155 23544 catalog_manager.cc:4841] TS 0be29894cca24c87b3d0f6f41bc6784b (127.12.222.195:36743): tablet c4a8778603da420abefc04652825ff8b (table test-table [id=85d13cfd5d2645718d860aaa53754a2d]) successfully deleted
W20260925 02:53:44.575778 23543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.222.193:37289 (error 108)
I20260925 02:53:44.575810 23542 catalog_manager.cc:4841] TS 4c410d5b60fa4646942b8aec94b75545 (127.12.222.194:36321): tablet c4a8778603da420abefc04652825ff8b (table test-table [id=85d13cfd5d2645718d860aaa53754a2d]) successfully deleted
I20260925 02:53:44.576011 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 23754
I20260925 02:53:44.577515 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:53:44.577175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:37289 (local address 127.0.0.1:60928)
0925 02:53:44.577339 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 02:53:44.577413 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.193:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260925 02:53:44.577667 23543 catalog_manager.cc:4584] TS c7c101c28ddb4395a39488ef5d8ae23c (127.12.222.193:37289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c4a8778603da420abefc04652825ff8b: Network error: Client connection negotiation failed: client connection to 127.12.222.193:37289: connect: Connection refused (error 111)
I20260925 02:53:44.581764 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 23889
I20260925 02:53:44.586853 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 23525
2026-09-25T02:53:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2338 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:53:44 dist-test-slave-252r krb5kdc[24066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:53:44 dist-test-slave-252r krb5kdc[24066](info): set up 2 sockets
Sep 25 02:53:44 dist-test-slave-252r krb5kdc[24066](info): commencing operation
krb5kdc: starting...
W20260925 02:53:45.629055 13179 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:53:45 dist-test-slave-252r krb5kdc[24066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:53:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:53:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.222.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.222.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.222.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/rangeradmin_127.12.222.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.222.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/rangeradmin_127.12.222.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.222.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.222.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.222.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/rangerlookup_127.12.222.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.222.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/rangerlookup_127.12.222.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.222.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.222.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.222.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/HTTP_127.12.222.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.222.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/HTTP_127.12.222.212@KRBTEST.COM.keytab.
I20260925 02:53:45.779302 13179 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskORQp8j/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/postgres -l logfile start
2026-09-25 02:53:48.130 UTC [24096] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 02:53:48.130 UTC [24096] LOG: listening on IPv4 address "127.12.222.212", port 55777
2026-09-25 02:53:48.133 UTC [24096] LOG: listening on Unix socket "/tmp/.s.PGSQL.55777"
2026-09-25 02:53:48.153 UTC [24099] LOG: database system was shut down at 2026-09-25 02:53:46 UTC
2026-09-25 02:53:48.156 UTC [24096] LOG: database system is ready to accept connections
I20260925 02:53:49.056300 13179 mini_postgres.cc:96] Postgres bound to 55777
2026-09-25 02:53:49.064 UTC [24107] FATAL: database "slave" does not exist
127.12.222.212:55777 - accepting connections
I20260925 02:53:49.065459 13179 mini_ranger.cc:161] Starting Ranger...
I20260925 02:53:49.081877 13179 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 02:53:49.501641 13179 mini_ranger.cc:87] Created ranger Postgres database
I20260925 02:53:49.501718 13179 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ranger-admin
2026-09-25 02:53:49,762 [I] DB FLAVOR :POSTGRES
2026-09-25 02:53:49,763 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:53:49,763 [I] Checking connection..
2026-09-25 02:53:49,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 02:53:50,006 [I] Checking connection passed.
2026-09-25 02:53:50,007 [I] --------- Verifying version history table ---------
2026-09-25 02:53:50,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:53:50,224 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:53:50,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:53:50,438 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:53:50,438 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 02:53:50,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 02:53:50.673 UTC [24171] WARNING: there is no transaction in progress
2026-09-25 02:53:50,679 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 02:53:50,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:53:50,919 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 02:53:50,919 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:53:50,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 02:53:51,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 02:53:51,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-252r.c.gcp-upstream.internal','N') ;"
2026-09-25 02:53:51,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:53:51,824 [I] Table x_portal_user does not exist in database ranger
2026-09-25 02:53:51,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:53:52,055 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 02:53:52,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:53:52,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:53:52,506 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 02:53:52,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 02:53:53.373 UTC [24281] WARNING: there is no transaction in progress
2026-09-25 02:53:53.388 UTC [24281] WARNING: there is no transaction in progress
2026-09-25 02:53:53.403 UTC [24281] WARNING: there is no transaction in progress
2026-09-25 02:53:53.417 UTC [24281] WARNING: there is no transaction in progress
2026-09-25 02:53:53.431 UTC [24281] WARNING: there is no transaction in progress
2026-09-25 02:53:53.448 UTC [24281] WARNING: there is no transaction in progress
2026-09-25 02:53:53.498 UTC [24281] WARNING: there is no transaction in progress
2026-09-25 02:53:53.504 UTC [24281] WARNING: there is no transaction in progress
2026-09-25 02:53:53.517 UTC [24281] WARNING: there is no transaction in progress
2026-09-25 02:53:53.526 UTC [24281] WARNING: there is no transaction in progress
2026-09-25 02:53:53.535 UTC [24281] WARNING: there is no transaction in progress
2026-09-25 02:53:53.548 UTC [24281] WARNING: there is no transaction in progress
2026-09-25 02:53:53.557 UTC [24281] WARNING: there is no transaction in progress
2026-09-25 02:53:53.562 UTC [24281] WARNING: there is no transaction in progress
2026-09-25 02:53:53.568 UTC [24281] WARNING: there is no transaction in progress
2026-09-25 02:53:53.574 UTC [24281] WARNING: there is no transaction in progress
2026-09-25 02:53:54,142 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 02:53:54,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 02:53:54,360 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 02:53:54,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:53:54,570 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 02:53:54,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:53:54,809 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 02:53:54,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:53:55,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:53:55,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-252r.c.gcp-upstream.internal';"
2026-09-25 02:53:55,461 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 02:53:55,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 02:53:55,696 [I] DB_PATCHES have already been applied
I20260925 02:53:55.701985 13179 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ranger-admin:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskORQp8j/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 02:53:55.702037 13179 mini_ranger.cc:193] Using host: 127.12.222.212
I20260925 02:53:55.705613 13179 mini_ranger.cc:227] Ranger admin URL: http://127.12.222.212:38417
Sep 25, 2026 2:53:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:53:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:53:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:53:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:53:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:53:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:53:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 02:53:56.760718 13179 mini_ranger.cc:160] Time spent starting Ranger: real 7.695s user 0.003s sys 0.006s
Sep 25, 2026 2:54:16 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:54:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 2:54:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:54:16 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:54:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:54:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:54:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:54:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:54:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:54:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:54:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:54:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:54:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:54:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:54:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:54:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:54:16 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:54:16 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 02:54:17.526580 13179 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.222.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.222.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.222.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/rangerkms_127.12.222.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.222.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/rangerkms_127.12.222.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.222.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.222.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.222.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/HTTP_127.12.222.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.222.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/HTTP_127.12.222.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 02:54:17 dist-test-slave-252r krb5kdc[24066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304857, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 02:54:17.608976 13179 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 02:54:17.618115 13179 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 02:54:18.400583 13179 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 02:54:18.400660 13179 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ranger-kms
I20260925 02:54:18.400755 13179 mini_ranger_kms.cc:227] Using postgres at 127.12.222.212:55777
2026-09-25 02:54:19,130 [I] DB FLAVOR :POSTGRES
2026-09-25 02:54:19,131 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:54:19,131 [I] Checking connection
2026-09-25 02:54:19,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 02:54:19,387 [I] connection success
2026-09-25 02:54:19,387 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 02:54:19,388 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 02:54:19,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 02:54:19,618 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 02:54:19,618 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:54:19,619 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 02:54:19,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:55777/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 02:54:19,876 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 02:54:20.323885 13179 mini_ranger_kms.cc:310] Created kms service
I20260925 02:54:20.482225 13179 mini_ranger_kms.cc:326] Created kudu user
I20260925 02:54:20.522294 13179 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 02:54:20 dist-test-slave-252r krb5kdc[24066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304857, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
I20260925 02:54:20.875725 13179 mini_ranger_kms.cc:384] Added ranger policy
I20260925 02:54:20.875866 13179 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ranger-kms:/tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 02:54:20.875890 13179 mini_ranger_kms.cc:240] Using host: 127.12.222.212
I20260925 02:54:20.879937 13179 mini_ranger_kms.cc:276] Ranger KMS PID: 24463
I20260925 02:54:20.880043 13179 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.222.212:44655
Sep 25, 2026 2:54:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:54:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:54:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:54:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:54:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:54:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 02:54:23.040557 13179 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.432s user 0.012s sys 0.280s
I20260925 02:54:23.040697 13179 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 02:54:23.040881 13179 mini_ranger_kms.cc:401] 127.12.222.212:44655/kms/v1/keys
Sep 25 02:54:23 dist-test-slave-252r krb5kdc[24066](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304863, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.222.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 02:54:23 dist-test-slave-252r krb5kdc[24066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304863, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.222.212@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
Sep 25 02:54:23 dist-test-slave-252r krb5kdc[24066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304863, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.222.212@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
Sep 25 02:54:25 dist-test-slave-252r krb5kdc[24066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:54:25.911319 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:43931
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:33669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:43931
--ranger_config_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.222.212:44655/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:54:26.039772 24514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:54:26.040042 24514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:54:26.040078 24514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:54:26.043825 24514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:54:26.043872 24514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:54:26.043890 24514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:54:26.043905 24514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:54:26.043918 24514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:54:26.048846 24514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:33669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.222.212:44655/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:43931
--ranger_config_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:43931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:54:26.049705 24514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:54:26.050808 24514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:54:26.058593 24519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:26.058651 24520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:26.058650 24522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:54:26.058702 24514 server_base.cc:1034] running on GCE node
I20260925 02:54:26.059329 24514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:54:26.060273 24514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:54:26.061498 24514 hybrid_clock.cc:648] HybridClock initialized: now 1790304866061457 us; error 43 us; skew 500 ppm
Sep 25 02:54:26 dist-test-slave-252r krb5kdc[24066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304866, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:54:26.070298 24514 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:54:26.071256 24514 webserver.cc:460] Webserver started at http://127.12.222.254:46013/ using document root <none> and password file <none>
I20260925 02:54:26.071758 24514 fs_manager.cc:359] Metadata directory not provided
I20260925 02:54:26.071839 24514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:54:26.072069 24514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:54:26 dist-test-slave-252r krb5kdc[24066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304866, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
I20260925 02:54:26.124116 24514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "b620b552ec5340c18cb9381c85930650"
format_stamp: "Formatted at 2026-09-25 02:54:26 on dist-test-slave-252r"
server_key: "42da25079bf6960b8d394118e37ea2ef"
server_key_iv: "238d667dd1362c43d90860fb57819cdb"
server_key_version: "kuduclusterkey@0"
I20260925 02:54:26.124750 24514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "b620b552ec5340c18cb9381c85930650"
format_stamp: "Formatted at 2026-09-25 02:54:26 on dist-test-slave-252r"
server_key: "42da25079bf6960b8d394118e37ea2ef"
server_key_iv: "238d667dd1362c43d90860fb57819cdb"
server_key_version: "kuduclusterkey@0"
I20260925 02:54:26.128480 24514 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:54:26.144541 24529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:26.146103 24514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:54:26.146245 24514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "b620b552ec5340c18cb9381c85930650"
format_stamp: "Formatted at 2026-09-25 02:54:26 on dist-test-slave-252r"
server_key: "42da25079bf6960b8d394118e37ea2ef"
server_key_iv: "238d667dd1362c43d90860fb57819cdb"
server_key_version: "kuduclusterkey@0"
I20260925 02:54:26.146368 24514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:54:26.161484 24514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:54:26.164011 24514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:54:26.164189 24514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:54:26.172618 24514 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:43931
I20260925 02:54:26.172667 24581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:43931 every 8 connection(s)
I20260925 02:54:26.173499 24514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:54:26.175948 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:54:26.178263 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 24514
I20260925 02:54:26.178376 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:54:26.178673 13179 external_mini_cluster.cc:1428] Setting key 68f00f2db1dcbc21a7136b32c95488c5
I20260925 02:54:26.181769 24582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650: Bootstrap starting.
I20260925 02:54:26.184146 24582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650: Neither blocks nor log segments found. Creating new log.
I20260925 02:54:26.184834 24582 log.cc:824] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650: Log is configured to *not* fsync() on all Append() calls
I20260925 02:54:26.187556 24582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650: No bootstrap required, opened a new log
Sep 25 02:54:26 dist-test-slave-252r krb5kdc[24066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:54:26.191160 24582 raft_consensus.cc:348] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b620b552ec5340c18cb9381c85930650" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43931 } }
I20260925 02:54:26.191358 24582 raft_consensus.cc:374] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:54:26.191391 24582 raft_consensus.cc:729] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b620b552ec5340c18cb9381c85930650, State: Initialized, Role: FOLLOWER
I20260925 02:54:26.191813 24582 consensus_queue.cc:260] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b620b552ec5340c18cb9381c85930650" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43931 } }
I20260925 02:54:26.191932 24582 raft_consensus.cc:388] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:54:26.192049 24582 raft_consensus.cc:482] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:54:26.192186 24582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:54:26.193007 24582 raft_consensus.cc:504] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b620b552ec5340c18cb9381c85930650" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43931 } }
I20260925 02:54:26.193399 24582 leader_election.cc:302] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b620b552ec5340c18cb9381c85930650; no voters:
I20260925 02:54:26.193727 24582 leader_election.cc:288] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:54:26.193795 24588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:54:26.193979 24588 raft_consensus.cc:686] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650 [term 1 LEADER]: Becoming Leader. State: Replica: b620b552ec5340c18cb9381c85930650, State: Running, Role: LEADER
I20260925 02:54:26.194216 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:26.180073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35888 (local address 127.12.222.254:43931)
0925 02:54:26.180490 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:26.180506 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:26.180547 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:54:26.181347 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:26.181370 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:26.181744 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:26.182096 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:26.182904 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:26.184183 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:26.185300 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:26.185786 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:26.185903 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:26.189777 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:26.189789 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:26.189801 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:26.189826 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:26.191328 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:26.191805 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:26.191815 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:26.191819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:26.191950 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:26.192259 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:26.192263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:26.192265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:26.192614 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:26.192812 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:26.193116 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 02:54:26.193374 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":136,"threads_started":1}
I20260925 02:54:26.194469 24588 consensus_queue.cc:237] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b620b552ec5340c18cb9381c85930650" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43931 } }
I20260925 02:54:26.194962 24582 sys_catalog.cc:564] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:54:26.196692 24589 sys_catalog.cc:455] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b620b552ec5340c18cb9381c85930650" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b620b552ec5340c18cb9381c85930650" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43931 } } }
I20260925 02:54:26.197072 24589 sys_catalog.cc:458] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650 [sys.catalog]: This master's current role is: LEADER
I20260925 02:54:26.197639 24582 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 02:54:26.197644 24590 sys_catalog.cc:455] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b620b552ec5340c18cb9381c85930650. Latest consensus state: current_term: 1 leader_uuid: "b620b552ec5340c18cb9381c85930650" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b620b552ec5340c18cb9381c85930650" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43931 } } }
I20260925 02:54:26.197796 24590 sys_catalog.cc:458] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650 [sys.catalog]: This master's current role is: LEADER
I20260925 02:54:26.198601 24603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:54:26.204380 24603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:54:26.209427 24603 catalog_manager.cc:1347] Generated new cluster ID: cdbe23e35e044dacb8f04cd640ae6210
I20260925 02:54:26.209478 24603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:54:26.234549 24603 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:54:26.235661 24603 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:54:26.244988 24603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650: Generated new TSK 0
I20260925 02:54:26.245659 24603 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
I20260925 02:54:26.863152 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.222.212:44655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:43931
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:33669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:54:26.991137 24628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:54:26.991379 24628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:54:26.991454 24628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:54:26.995620 24628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:54:26.995682 24628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:54:26.995755 24628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:54:27.001302 24628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:33669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.222.212:44655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:43931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:54:27.003111 24628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:54:27.004354 24628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:54:27.013427 24636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:27.013512 24638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:27.013511 24635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:54:27.013790 24628 server_base.cc:1034] running on GCE node
I20260925 02:54:27.014254 24628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:54:27.014789 24628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:54:27.015969 24628 hybrid_clock.cc:648] HybridClock initialized: now 1790304867015947 us; error 33 us; skew 500 ppm
Sep 25 02:54:27 dist-test-slave-252r krb5kdc[24066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304867, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:54:27.018740 24628 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:54:27.019762 24628 webserver.cc:460] Webserver started at http://127.12.222.193:37323/ using document root <none> and password file <none>
I20260925 02:54:27.020443 24628 fs_manager.cc:359] Metadata directory not provided
I20260925 02:54:27.020553 24628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:54:27.020771 24628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:54:27 dist-test-slave-252r krb5kdc[24066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304867, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
I20260925 02:54:27.040987 24628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "b52d02f15dc64af3a250e743ed4f6d53"
format_stamp: "Formatted at 2026-09-25 02:54:27 on dist-test-slave-252r"
server_key: "1882d4238fdd94d6f99995e124a50065"
server_key_iv: "0d1efd350cb13213e1095bbdb71ad610"
server_key_version: "kuduclusterkey@0"
I20260925 02:54:27.041769 24628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "b52d02f15dc64af3a250e743ed4f6d53"
format_stamp: "Formatted at 2026-09-25 02:54:27 on dist-test-slave-252r"
server_key: "1882d4238fdd94d6f99995e124a50065"
server_key_iv: "0d1efd350cb13213e1095bbdb71ad610"
server_key_version: "kuduclusterkey@0"
I20260925 02:54:27.057685 24628 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.000s sys 0.003s
I20260925 02:54:27.074218 24646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:27.075598 24628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:54:27.075709 24628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "b52d02f15dc64af3a250e743ed4f6d53"
format_stamp: "Formatted at 2026-09-25 02:54:27 on dist-test-slave-252r"
server_key: "1882d4238fdd94d6f99995e124a50065"
server_key_iv: "0d1efd350cb13213e1095bbdb71ad610"
server_key_version: "kuduclusterkey@0"
I20260925 02:54:27.075809 24628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:54:27.092903 24628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:54:27.096113 24628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:54:27.096328 24628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:54:27.096977 24628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:54:27.097882 24628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:54:27.097976 24628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:27.098080 24628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:54:27.098151 24628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:27.111729 24628 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:41763
I20260925 02:54:27.112943 24628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:54:27.114027 24760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:41763 every 8 connection(s)
I20260925 02:54:27.121019 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 24628
I20260925 02:54:27.121146 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:54:27.121454 13179 external_mini_cluster.cc:1428] Setting key 32a8fe09a5f7befcd3b3bfcb0e8f2a4f
Sep 25 02:54:27 dist-test-slave-252r krb5kdc[24066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304867, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:54:27.134711 24764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:27.116691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43931 (local address 127.12.222.193:39627)
0925 02:54:27.120626 (+ 3935us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:27.120679 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:27.122086 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:27.122926 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:27.122935 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:27.123241 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:54:27.123870 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:27.123890 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.125043 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.125047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:27.126236 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:27.126246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.126959 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.126963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:27.127139 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:27.127741 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:54:27.127758 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:54:27.129269 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:54:27.131690 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:54:27.131695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:54:27.131708 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:54:27.132041 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:54:27.132435 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:54:27.132440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:54:27.132443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:54:27.132619 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:54:27.133104 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:54:27.133112 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:54:27.133379 (+ 267us) client_negotiation.cc:769] Sending connection context
0925 02:54:27.133562 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 02:54:27.133866 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3695,"thread_start_us":215,"threads_started":1}
I20260925 02:54:27.135349 24763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:27.116271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:39627 (local address 127.12.222.254:43931)
0925 02:54:27.122470 (+ 6199us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:27.122478 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:27.122490 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:54:27.122522 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:27.122525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:27.122569 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:27.122640 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:27.124032 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.124889 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.126366 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.126699 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.126792 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:27.129434 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:27.129437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:27.129440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:27.129461 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:27.131557 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:27.132163 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:27.132167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:27.132169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:27.132322 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:27.132734 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:27.132737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:27.132741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:27.132961 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:27.133051 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:27.135084 (+ 2033us) server_negotiation.cc:299] Negotiation successful
0925 02:54:27.135194 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6128,"thread_start_us":113,"threads_started":1}
I20260925 02:54:27.136790 24761 heartbeater.cc:347] Connected to a master server at 127.12.222.254:43931
I20260925 02:54:27.137040 24761 heartbeater.cc:464] Registering TS with master...
I20260925 02:54:27.137565 24761 heartbeater.cc:511] Master 127.12.222.254:43931 requested a full tablet report, sending...
I20260925 02:54:27.139151 24546 ts_manager.cc:194] Registered new tserver with Master: b52d02f15dc64af3a250e743ed4f6d53 (127.12.222.193:41763)
I20260925 02:54:27.140194 24546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:39627
WARNING: no policy specified for kudu/127.12.222.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
I20260925 02:54:27.168511 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.222.212:44655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:43931
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:33669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:54:27 dist-test-slave-252r krb5kdc[24066](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304867, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 02:54:27.316128 24767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:54:27.316421 24767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:54:27.316527 24767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:54:27.320919 24767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:54:27.321019 24767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:54:27.321132 24767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:54:27.326316 24767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:33669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.222.212:44655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:43931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:54:27.327266 24767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:54:27.328207 24767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:54:27.339154 24776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:27.340651 24774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:27.340997 24773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:54:27.341486 24767 server_base.cc:1034] running on GCE node
I20260925 02:54:27.341878 24767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:54:27.342545 24767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:54:27.343729 24767 hybrid_clock.cc:648] HybridClock initialized: now 1790304867343690 us; error 90 us; skew 500 ppm
Sep 25 02:54:27 dist-test-slave-252r krb5kdc[24066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304867, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:54:27.346727 24767 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:54:27.347622 24767 webserver.cc:460] Webserver started at http://127.12.222.194:46097/ using document root <none> and password file <none>
I20260925 02:54:27.348155 24767 fs_manager.cc:359] Metadata directory not provided
I20260925 02:54:27.348258 24767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:54:27.348465 24767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:54:27 dist-test-slave-252r krb5kdc[24066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304867, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
I20260925 02:54:27.369155 24767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "5061a83b49114b6ba65c9ea0d77f3d38"
format_stamp: "Formatted at 2026-09-25 02:54:27 on dist-test-slave-252r"
server_key: "cc259958c637836cfcd082f1dd987554"
server_key_iv: "0580f7357cd7817b7717c436dd9b5340"
server_key_version: "kuduclusterkey@0"
I20260925 02:54:27.369943 24767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "5061a83b49114b6ba65c9ea0d77f3d38"
format_stamp: "Formatted at 2026-09-25 02:54:27 on dist-test-slave-252r"
server_key: "cc259958c637836cfcd082f1dd987554"
server_key_iv: "0580f7357cd7817b7717c436dd9b5340"
server_key_version: "kuduclusterkey@0"
I20260925 02:54:27.377605 24767 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260925 02:54:27.401386 24783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:27.402980 24767 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260925 02:54:27.403084 24767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "5061a83b49114b6ba65c9ea0d77f3d38"
format_stamp: "Formatted at 2026-09-25 02:54:27 on dist-test-slave-252r"
server_key: "cc259958c637836cfcd082f1dd987554"
server_key_iv: "0580f7357cd7817b7717c436dd9b5340"
server_key_version: "kuduclusterkey@0"
I20260925 02:54:27.403180 24767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:54:27.420219 24767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:54:27.423594 24767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:54:27.423801 24767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:54:27.424468 24767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:54:27.425443 24767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:54:27.425542 24767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:27.425637 24767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:54:27.425709 24767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:27.435748 24767 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:36607
I20260925 02:54:27.439450 24767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:54:27.440534 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 24767
I20260925 02:54:27.440647 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:54:27.441002 13179 external_mini_cluster.cc:1428] Setting key e60fb372ec1da946d6faa8dbf7b25f7e
I20260925 02:54:27.445147 24898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:36607 every 8 connection(s)
Sep 25 02:54:27 dist-test-slave-252r krb5kdc[24066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304867, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.222.195@KRBTEST.COM; defaulting to no policy
I20260925 02:54:27.459132 24763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:27.447266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:36131 (local address 127.12.222.254:43931)
0925 02:54:27.447423 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:27.447428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:27.447725 (+ 297us) server_negotiation.cc:407] Connection header received
0925 02:54:27.448776 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:27.448779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:27.448825 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:27.448895 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:27.450294 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.451125 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.452174 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.452602 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.452711 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:27.455539 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:27.455551 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:27.455554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:27.455583 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:27.457258 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:27.457728 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:27.457732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:27.457734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:27.457834 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:27.458143 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:27.458149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:27.458151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:27.458313 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:27.458420 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:27.458862 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 02:54:27.458978 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.195@KRBTEST.COM" created.
I20260925 02:54:27.461169 24901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:27.445914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43931 (local address 127.12.222.194:36131)
0925 02:54:27.447582 (+ 1668us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:27.447637 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:27.448585 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:27.449173 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:27.449183 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:27.449519 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:54:27.450126 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:27.450147 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.451244 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.451247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:27.452047 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:27.452055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.452876 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.452882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:27.453140 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:27.453729 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:54:27.453747 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:54:27.455316 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:54:27.457389 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:54:27.457394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:54:27.457406 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:54:27.457627 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:54:27.457918 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:54:27.457922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:54:27.457924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:54:27.458054 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:54:27.458444 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:54:27.458449 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:54:27.458654 (+ 205us) client_negotiation.cc:769] Sending connection context
0925 02:54:27.458823 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 02:54:27.459106 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1450,"thread_start_us":122,"threads_started":1}
I20260925 02:54:27.462235 24899 heartbeater.cc:347] Connected to a master server at 127.12.222.254:43931
I20260925 02:54:27.462452 24899 heartbeater.cc:464] Registering TS with master...
I20260925 02:54:27.462872 24899 heartbeater.cc:511] Master 127.12.222.254:43931 requested a full tablet report, sending...
I20260925 02:54:27.463809 24546 ts_manager.cc:194] Registered new tserver with Master: 5061a83b49114b6ba65c9ea0d77f3d38 (127.12.222.194:36607)
I20260925 02:54:27.464447 24546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:36131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
I20260925 02:54:27.493857 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.222.212:44655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:43931
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:33669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:54:27 dist-test-slave-252r krb5kdc[24066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304867, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
W20260925 02:54:27.649528 24904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:54:27.649797 24904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:54:27.649858 24904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:54:27.655388 24904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:54:27.655450 24904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:54:27.655531 24904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:54:27.662375 24904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:33669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.222.212:44655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:43931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:54:27.663480 24904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:54:27.664587 24904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:54:27.680891 24910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:27.681332 24909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:54:27.681514 24912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:54:27.682639 24904 server_base.cc:1034] running on GCE node
I20260925 02:54:27.683156 24904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:54:27.683841 24904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:54:27.687105 24904 hybrid_clock.cc:648] HybridClock initialized: now 1790304867687037 us; error 71 us; skew 500 ppm
Sep 25 02:54:27 dist-test-slave-252r krb5kdc[24066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304867, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:54:27.690536 24904 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:54:27.691511 24904 webserver.cc:460] Webserver started at http://127.12.222.195:45033/ using document root <none> and password file <none>
I20260925 02:54:27.692155 24904 fs_manager.cc:359] Metadata directory not provided
I20260925 02:54:27.692219 24904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:54:27.692507 24904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:54:27 dist-test-slave-252r krb5kdc[24066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304867, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
I20260925 02:54:27.709116 24904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "6b771602d6b7413f8e004daee2b53355"
format_stamp: "Formatted at 2026-09-25 02:54:27 on dist-test-slave-252r"
server_key: "c913e2866376059834627f19311ed543"
server_key_iv: "e6bfe8498237a165c05c6f8032815043"
server_key_version: "kuduclusterkey@0"
I20260925 02:54:27.709762 24904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "6b771602d6b7413f8e004daee2b53355"
format_stamp: "Formatted at 2026-09-25 02:54:27 on dist-test-slave-252r"
server_key: "c913e2866376059834627f19311ed543"
server_key_iv: "e6bfe8498237a165c05c6f8032815043"
server_key_version: "kuduclusterkey@0"
I20260925 02:54:27.722083 24904 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.005s sys 0.000s
I20260925 02:54:27.740581 24919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:27.742187 24904 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260925 02:54:27.742338 24904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "6b771602d6b7413f8e004daee2b53355"
format_stamp: "Formatted at 2026-09-25 02:54:27 on dist-test-slave-252r"
server_key: "c913e2866376059834627f19311ed543"
server_key_iv: "e6bfe8498237a165c05c6f8032815043"
server_key_version: "kuduclusterkey@0"
I20260925 02:54:27.742455 24904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:54:27.768769 24904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:54:27.771957 24904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:54:27.772125 24904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:54:27.772809 24904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:54:27.773705 24904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:54:27.773764 24904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:27.773814 24904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:54:27.773852 24904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:54:27.786777 24904 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:38985
I20260925 02:54:27.787813 24904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:54:27.788592 25033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:38985 every 8 connection(s)
I20260925 02:54:27.796602 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 24904
I20260925 02:54:27.796727 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:54:27.797201 13179 external_mini_cluster.cc:1428] Setting key e339c8ac495c2fb21e4855331b34ff69
Sep 25 02:54:27 dist-test-slave-252r krb5kdc[24066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304867, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:54:27.808874 24763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:27.792121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:35793 (local address 127.12.222.254:43931)
0925 02:54:27.792296 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:27.792301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:27.792847 (+ 546us) server_negotiation.cc:407] Connection header received
0925 02:54:27.794031 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:27.794035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:27.794086 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:27.794133 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:27.795967 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.798325 (+ 2358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.800468 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.800739 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.800827 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:27.804070 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:27.804073 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:27.804076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:27.804102 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:27.806455 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:27.807060 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:27.807064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:27.807066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:27.807172 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:27.807611 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:27.807614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:27.807615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:27.807829 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:27.807894 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:27.808612 (+ 718us) server_negotiation.cc:299] Negotiation successful
0925 02:54:27.808718 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:54:27.811414 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:27.791151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43931 (local address 127.12.222.195:35793)
0925 02:54:27.792681 (+ 1530us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:27.792736 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:27.793794 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:27.794550 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:27.794558 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:27.794885 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:54:27.795782 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:27.795799 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.798882 (+ 3083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.798885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:27.800335 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:27.800346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.801020 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.801023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:27.801280 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:27.802024 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:54:27.802050 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:54:27.803882 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:54:27.806590 (+ 2708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:54:27.806595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:54:27.806609 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:54:27.806950 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:54:27.807328 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:54:27.807331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:54:27.807333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:54:27.807500 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:54:27.807982 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:54:27.807988 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:54:27.808284 (+ 296us) client_negotiation.cc:769] Sending connection context
0925 02:54:27.809299 (+ 1015us) client_negotiation.cc:240] Negotiation successful
0925 02:54:27.809582 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1324,"thread_start_us":136,"threads_started":1}
I20260925 02:54:27.812800 25034 heartbeater.cc:347] Connected to a master server at 127.12.222.254:43931
I20260925 02:54:27.813006 25034 heartbeater.cc:464] Registering TS with master...
I20260925 02:54:27.813516 25034 heartbeater.cc:511] Master 127.12.222.254:43931 requested a full tablet report, sending...
I20260925 02:54:27.814499 24546 ts_manager.cc:194] Registered new tserver with Master: 6b771602d6b7413f8e004daee2b53355 (127.12.222.195:38985)
I20260925 02:54:27.815083 24546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:35793
I20260925 02:54:27.822340 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:54:27.833065 24763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:27.824519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38382 (local address 127.12.222.254:43931)
0925 02:54:27.824619 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:27.824623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:27.825208 (+ 585us) server_negotiation.cc:407] Connection header received
0925 02:54:27.825371 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:27.825374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:27.825412 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:27.825453 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:54:27.826398 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.827091 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.828090 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.828392 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.828454 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:27.829952 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:54:27.829956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:54:27.829958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:54:27.829979 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:54:27.831424 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:27.831877 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:27.831880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:27.831882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:27.831956 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:54:27.832221 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:54:27.832253 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:54:27.832254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:54:27.832447 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:54:27.832533 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:27.832849 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 02:54:27.832941 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:54:27.836099 24546 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:54:27.840165 24546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:54:27.864480 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:27.850430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:41763 (local address 127.0.0.1:60630)
0925 02:54:27.853751 (+ 3321us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:27.853762 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:27.853844 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:27.855571 (+ 1727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:27.855574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:27.855590 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:54:27.855768 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:27.855775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.857167 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.857170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:27.858492 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:27.858502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.863941 (+ 5439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.863944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:27.864066 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:27.864115 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 02:54:27.864215 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 02:54:27.864347 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3268,"thread_start_us":80,"threads_started":1}
I20260925 02:54:27.865422 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:27.850586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60630 (local address 127.12.222.193:41763)
0925 02:54:27.852616 (+ 2030us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:27.852623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:27.853928 (+ 1305us) server_negotiation.cc:407] Connection header received
0925 02:54:27.854001 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:27.854006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:27.854171 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:27.854289 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:54:27.855900 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.857047 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.858621 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.860132 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.861262 (+ 1130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:27.861401 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:27.864763 (+ 3362us) server_negotiation.cc:299] Negotiation successful
0925 02:54:27.864865 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1902,"thread_start_us":81,"threads_started":1}
I20260925 02:54:27.867245 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:27.848927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:36607 (local address 127.0.0.1:51014)
0925 02:54:27.851598 (+ 2671us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:27.851645 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:27.851761 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:27.853417 (+ 1656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:27.853422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:27.853445 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:54:27.853563 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:27.853569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.861478 (+ 7909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.861481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:27.862698 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:27.862708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.866914 (+ 4206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.866918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:27.866997 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:27.867017 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 02:54:27.867079 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 02:54:27.867138 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2507,"thread_start_us":88,"threads_started":1}
I20260925 02:54:27.867731 25047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:27.849713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51014 (local address 127.12.222.194:36607)
0925 02:54:27.852797 (+ 3084us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:27.852807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:27.852840 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:54:27.852914 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:27.852919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:27.853111 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:27.853259 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:54:27.854423 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.861136 (+ 6713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.862865 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.866642 (+ 3777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.866747 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:27.866857 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:27.867487 (+ 630us) server_negotiation.cc:299] Negotiation successful
0925 02:54:27.867584 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2996,"thread_start_us":117,"threads_started":1}
I20260925 02:54:27.868422 24695 tablet_service.cc:1467] Processing CreateTablet for tablet e1dd0798feb94001874f561db7b266dc (DEFAULT_TABLE table=test-table [id=9b521f8ce764418bb30c1fc4ce472b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:54:27.869529 24695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1dd0798feb94001874f561db7b266dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:54:27.870249 24832 tablet_service.cc:1467] Processing CreateTablet for tablet e1dd0798feb94001874f561db7b266dc (DEFAULT_TABLE table=test-table [id=9b521f8ce764418bb30c1fc4ce472b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:54:27.871189 24832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1dd0798feb94001874f561db7b266dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:54:27.879988 25052 tablet_bootstrap.cc:492] T e1dd0798feb94001874f561db7b266dc P b52d02f15dc64af3a250e743ed4f6d53: Bootstrap starting.
I20260925 02:54:27.881239 25053 tablet_bootstrap.cc:492] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38: Bootstrap starting.
I20260925 02:54:27.883781 25053 tablet_bootstrap.cc:654] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38: Neither blocks nor log segments found. Creating new log.
I20260925 02:54:27.884380 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:27.851107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:38985 (local address 127.0.0.1:42088)
0925 02:54:27.864394 (+ 13287us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:27.864407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:27.864512 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:27.874990 (+ 10478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:27.874994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:27.875014 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:54:27.875106 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:27.875112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.880163 (+ 5051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.880167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:27.882189 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:27.882201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.883962 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.883965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:27.884049 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:27.884071 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:54:27.884141 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 02:54:27.884218 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13228,"thread_start_us":96,"threads_started":1}
I20260925 02:54:27.884449 25051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:27.851245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42088 (local address 127.12.222.195:38985)
0925 02:54:27.857177 (+ 5932us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:27.857190 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:27.864485 (+ 7295us) server_negotiation.cc:407] Connection header received
0925 02:54:27.864880 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:27.864888 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:27.865060 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:27.865235 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:54:27.875226 (+ 9991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.876407 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.882348 (+ 5941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:27.883095 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:27.883217 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:27.883340 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:27.884189 (+ 849us) server_negotiation.cc:299] Negotiation successful
0925 02:54:27.884324 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5812,"thread_start_us":130,"threads_started":1}
I20260925 02:54:27.884582 25053 log.cc:824] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38: Log is configured to *not* fsync() on all Append() calls
I20260925 02:54:27.886520 25052 tablet_bootstrap.cc:654] T e1dd0798feb94001874f561db7b266dc P b52d02f15dc64af3a250e743ed4f6d53: Neither blocks nor log segments found. Creating new log.
I20260925 02:54:27.887240 25052 log.cc:824] T e1dd0798feb94001874f561db7b266dc P b52d02f15dc64af3a250e743ed4f6d53: Log is configured to *not* fsync() on all Append() calls
I20260925 02:54:27.887255 24968 tablet_service.cc:1467] Processing CreateTablet for tablet e1dd0798feb94001874f561db7b266dc (DEFAULT_TABLE table=test-table [id=9b521f8ce764418bb30c1fc4ce472b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:54:27.888172 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1dd0798feb94001874f561db7b266dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:54:27.889292 25053 tablet_bootstrap.cc:492] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38: No bootstrap required, opened a new log
I20260925 02:54:27.889465 25053 ts_tablet_manager.cc:1397] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260925 02:54:27.893815 25053 raft_consensus.cc:348] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5061a83b49114b6ba65c9ea0d77f3d38" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36607 } } peers { permanent_uuid: "b52d02f15dc64af3a250e743ed4f6d53" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41763 } } peers { permanent_uuid: "6b771602d6b7413f8e004daee2b53355" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38985 } }
I20260925 02:54:27.894127 25053 raft_consensus.cc:374] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:54:27.894189 25053 raft_consensus.cc:729] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5061a83b49114b6ba65c9ea0d77f3d38, State: Initialized, Role: FOLLOWER
I20260925 02:54:27.894632 25053 consensus_queue.cc:260] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5061a83b49114b6ba65c9ea0d77f3d38" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36607 } } peers { permanent_uuid: "b52d02f15dc64af3a250e743ed4f6d53" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41763 } } peers { permanent_uuid: "6b771602d6b7413f8e004daee2b53355" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38985 } }
I20260925 02:54:27.895677 25053 ts_tablet_manager.cc:1428] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 02:54:27.896014 24899 heartbeater.cc:503] Master 127.12.222.254:43931 was elected leader, sending a full tablet report...
I20260925 02:54:27.900672 25056 tablet_bootstrap.cc:492] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355: Bootstrap starting.
I20260925 02:54:27.903714 25056 tablet_bootstrap.cc:654] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355: Neither blocks nor log segments found. Creating new log.
I20260925 02:54:27.904479 25056 log.cc:824] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355: Log is configured to *not* fsync() on all Append() calls
I20260925 02:54:27.905833 25052 tablet_bootstrap.cc:492] T e1dd0798feb94001874f561db7b266dc P b52d02f15dc64af3a250e743ed4f6d53: No bootstrap required, opened a new log
I20260925 02:54:27.906016 25052 ts_tablet_manager.cc:1397] T e1dd0798feb94001874f561db7b266dc P b52d02f15dc64af3a250e743ed4f6d53: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20260925 02:54:27.910516 25052 raft_consensus.cc:348] T e1dd0798feb94001874f561db7b266dc P b52d02f15dc64af3a250e743ed4f6d53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5061a83b49114b6ba65c9ea0d77f3d38" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36607 } } peers { permanent_uuid: "b52d02f15dc64af3a250e743ed4f6d53" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41763 } } peers { permanent_uuid: "6b771602d6b7413f8e004daee2b53355" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38985 } }
I20260925 02:54:27.910765 25052 raft_consensus.cc:374] T e1dd0798feb94001874f561db7b266dc P b52d02f15dc64af3a250e743ed4f6d53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:54:27.910816 25052 raft_consensus.cc:729] T e1dd0798feb94001874f561db7b266dc P b52d02f15dc64af3a250e743ed4f6d53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52d02f15dc64af3a250e743ed4f6d53, State: Initialized, Role: FOLLOWER
I20260925 02:54:27.911269 25052 consensus_queue.cc:260] T e1dd0798feb94001874f561db7b266dc P b52d02f15dc64af3a250e743ed4f6d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5061a83b49114b6ba65c9ea0d77f3d38" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36607 } } peers { permanent_uuid: "b52d02f15dc64af3a250e743ed4f6d53" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41763 } } peers { permanent_uuid: "6b771602d6b7413f8e004daee2b53355" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38985 } }
I20260925 02:54:27.912209 25052 ts_tablet_manager.cc:1428] T e1dd0798feb94001874f561db7b266dc P b52d02f15dc64af3a250e743ed4f6d53: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 02:54:27.912549 24761 heartbeater.cc:503] Master 127.12.222.254:43931 was elected leader, sending a full tablet report...
I20260925 02:54:27.917514 25056 tablet_bootstrap.cc:492] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355: No bootstrap required, opened a new log
I20260925 02:54:27.917698 25056 ts_tablet_manager.cc:1397] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260925 02:54:27.921185 25056 raft_consensus.cc:348] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5061a83b49114b6ba65c9ea0d77f3d38" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36607 } } peers { permanent_uuid: "b52d02f15dc64af3a250e743ed4f6d53" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41763 } } peers { permanent_uuid: "6b771602d6b7413f8e004daee2b53355" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38985 } }
I20260925 02:54:27.921427 25056 raft_consensus.cc:374] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:54:27.921490 25056 raft_consensus.cc:729] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b771602d6b7413f8e004daee2b53355, State: Initialized, Role: FOLLOWER
I20260925 02:54:27.921929 25056 consensus_queue.cc:260] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5061a83b49114b6ba65c9ea0d77f3d38" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36607 } } peers { permanent_uuid: "b52d02f15dc64af3a250e743ed4f6d53" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41763 } } peers { permanent_uuid: "6b771602d6b7413f8e004daee2b53355" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38985 } }
I20260925 02:54:27.922812 25056 ts_tablet_manager.cc:1428] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 02:54:27.923170 25034 heartbeater.cc:503] Master 127.12.222.254:43931 was elected leader, sending a full tablet report...
W20260925 02:54:27.945019 24900 tablet.cc:2369] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:54:28.039611 25035 tablet.cc:2369] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 02:54:28 dist-test-slave-252r krb5kdc[24066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304867, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
I20260925 02:54:28.075155 25057 raft_consensus.cc:482] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:54:28.075383 25057 raft_consensus.cc:504] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5061a83b49114b6ba65c9ea0d77f3d38" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36607 } } peers { permanent_uuid: "b52d02f15dc64af3a250e743ed4f6d53" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41763 } } peers { permanent_uuid: "6b771602d6b7413f8e004daee2b53355" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38985 } }
I20260925 02:54:28.076866 25057 leader_election.cc:288] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b52d02f15dc64af3a250e743ed4f6d53 (127.12.222.193:41763), 6b771602d6b7413f8e004daee2b53355 (127.12.222.195:38985)
I20260925 02:54:28.082723 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:28.077666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:32873 (local address 127.12.222.193:41763)
0925 02:54:28.077807 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:28.077820 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:28.078358 (+ 538us) server_negotiation.cc:407] Connection header received
0925 02:54:28.078559 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:28.078562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:28.078608 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:28.078652 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:54:28.079047 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:28.079807 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:28.081447 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:28.082066 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:28.082155 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:28.082205 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:28.082534 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 02:54:28.082601 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:54:28.083310 25062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:28.077201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:41763 (local address 127.12.222.194:32873)
0925 02:54:28.078278 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:28.078291 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:28.078439 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:28.078790 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:28.078792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:28.078809 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:54:28.078922 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:28.078929 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:28.079966 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:28.079969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:28.081314 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:28.081325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:28.082308 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:28.082311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:28.082411 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:28.082431 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 02:54:28.083088 (+ 657us) client_negotiation.cc:240] Negotiation successful
0925 02:54:28.083162 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"thread_start_us":95,"threads_started":1}
I20260925 02:54:28.084074 24715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dd0798feb94001874f561db7b266dc" candidate_uuid: "5061a83b49114b6ba65c9ea0d77f3d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52d02f15dc64af3a250e743ed4f6d53" is_pre_election: true
I20260925 02:54:28.084400 24715 raft_consensus.cc:2454] T e1dd0798feb94001874f561db7b266dc P b52d02f15dc64af3a250e743ed4f6d53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5061a83b49114b6ba65c9ea0d77f3d38 in term 0.
I20260925 02:54:28.084858 24786 leader_election.cc:302] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5061a83b49114b6ba65c9ea0d77f3d38, b52d02f15dc64af3a250e743ed4f6d53; no voters:
I20260925 02:54:28.085114 25057 raft_consensus.cc:2781] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:54:28.085201 25057 raft_consensus.cc:482] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:54:28.085240 25057 raft_consensus.cc:3037] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:54:28.086575 25057 raft_consensus.cc:504] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5061a83b49114b6ba65c9ea0d77f3d38" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36607 } } peers { permanent_uuid: "b52d02f15dc64af3a250e743ed4f6d53" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41763 } } peers { permanent_uuid: "6b771602d6b7413f8e004daee2b53355" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38985 } }
I20260925 02:54:28.086943 25057 leader_election.cc:288] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [CANDIDATE]: Term 1 election: Requested vote from peers b52d02f15dc64af3a250e743ed4f6d53 (127.12.222.193:41763), 6b771602d6b7413f8e004daee2b53355 (127.12.222.195:38985)
I20260925 02:54:28.087735 24715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dd0798feb94001874f561db7b266dc" candidate_uuid: "5061a83b49114b6ba65c9ea0d77f3d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52d02f15dc64af3a250e743ed4f6d53"
I20260925 02:54:28.087790 25051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:28.078156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:56339 (local address 127.12.222.195:38985)
0925 02:54:28.078267 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:28.078274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:28.083933 (+ 5659us) server_negotiation.cc:407] Connection header received
0925 02:54:28.084030 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:28.084033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:28.084076 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:28.084154 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:54:28.084444 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:28.085053 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:28.086504 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:28.087118 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:28.087205 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:28.087249 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:28.087612 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 02:54:28.087683 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 02:54:28.087848 24715 raft_consensus.cc:3037] T e1dd0798feb94001874f561db7b266dc P b52d02f15dc64af3a250e743ed4f6d53 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:54:28.089214 25062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:28.078029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:38985 (local address 127.12.222.194:56339)
0925 02:54:28.083855 (+ 5826us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:28.083868 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:28.083955 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:28.084178 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:54:28.084180 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:54:28.084195 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:54:28.084279 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:28.084284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:28.085236 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:28.085238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:28.086361 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:54:28.086372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:54:28.087352 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:54:28.087355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:54:28.087453 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:28.087490 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 02:54:28.089040 (+ 1550us) client_negotiation.cc:240] Negotiation successful
0925 02:54:28.089104 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5768,"thread_start_us":50,"threads_started":1}
I20260925 02:54:28.089576 24715 raft_consensus.cc:2454] T e1dd0798feb94001874f561db7b266dc P b52d02f15dc64af3a250e743ed4f6d53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5061a83b49114b6ba65c9ea0d77f3d38 in term 1.
I20260925 02:54:28.089967 24786 leader_election.cc:302] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5061a83b49114b6ba65c9ea0d77f3d38, b52d02f15dc64af3a250e743ed4f6d53; no voters:
I20260925 02:54:28.090442 24985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dd0798feb94001874f561db7b266dc" candidate_uuid: "5061a83b49114b6ba65c9ea0d77f3d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b771602d6b7413f8e004daee2b53355"
I20260925 02:54:28.090490 24972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dd0798feb94001874f561db7b266dc" candidate_uuid: "5061a83b49114b6ba65c9ea0d77f3d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b771602d6b7413f8e004daee2b53355" is_pre_election: true
I20260925 02:54:28.090637 24985 raft_consensus.cc:3037] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:54:28.091572 25057 raft_consensus.cc:2781] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:54:28.091754 25057 raft_consensus.cc:686] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [term 1 LEADER]: Becoming Leader. State: Replica: 5061a83b49114b6ba65c9ea0d77f3d38, State: Running, Role: LEADER
I20260925 02:54:28.091836 24985 raft_consensus.cc:2454] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5061a83b49114b6ba65c9ea0d77f3d38 in term 1.
I20260925 02:54:28.092082 25057 consensus_queue.cc:237] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5061a83b49114b6ba65c9ea0d77f3d38" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36607 } } peers { permanent_uuid: "b52d02f15dc64af3a250e743ed4f6d53" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41763 } } peers { permanent_uuid: "6b771602d6b7413f8e004daee2b53355" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38985 } }
I20260925 02:54:28.095235 24545 catalog_manager.cc:5495] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5061a83b49114b6ba65c9ea0d77f3d38 (127.12.222.194). New cstate: current_term: 1 leader_uuid: "5061a83b49114b6ba65c9ea0d77f3d38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5061a83b49114b6ba65c9ea0d77f3d38" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52d02f15dc64af3a250e743ed4f6d53" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b771602d6b7413f8e004daee2b53355" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38985 } health_report { overall_health: UNKNOWN } } }
W20260925 02:54:28.115727 24762 tablet.cc:2369] T e1dd0798feb94001874f561db7b266dc P b52d02f15dc64af3a250e743ed4f6d53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:54:28.155949 25047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:28.150945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51016 (local address 127.12.222.194:36607)
0925 02:54:28.151111 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 02:54:28.151116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:54:28.151209 (+ 93us) server_negotiation.cc:407] Connection header received
0925 02:54:28.151387 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:54:28.151391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:54:28.151459 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:54:28.151563 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:54:28.151982 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:28.152798 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:28.154041 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:54:28.154368 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:54:28.154452 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:54:28.154834 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:54:28.155311 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:54:28.155420 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:54:28.155757 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 02:54:28.155822 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:54:28.163725 24985 raft_consensus.cc:1261] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355 [term 1 FOLLOWER]: Refusing update from remote peer 5061a83b49114b6ba65c9ea0d77f3d38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:54:28.163866 24715 raft_consensus.cc:1261] T e1dd0798feb94001874f561db7b266dc P b52d02f15dc64af3a250e743ed4f6d53 [term 1 FOLLOWER]: Refusing update from remote peer 5061a83b49114b6ba65c9ea0d77f3d38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:54:28.164345 25057 consensus_queue.cc:1035] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b771602d6b7413f8e004daee2b53355" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 38985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:54:28.164976 25057 consensus_queue.cc:1035] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52d02f15dc64af3a250e743ed4f6d53" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:54:28.185932 24546 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:54:28.186082 24546 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:54:28.188652 24546 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b620b552ec5340c18cb9381c85930650: Sending DeleteTablet for 3 replicas of tablet e1dd0798feb94001874f561db7b266dc
I20260925 02:54:28.189479 24968 tablet_service.cc:1514] Processing DeleteTablet for tablet e1dd0798feb94001874f561db7b266dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:54:28 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:42088
I20260925 02:54:28.190840 24695 tablet_service.cc:1514] Processing DeleteTablet for tablet e1dd0798feb94001874f561db7b266dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:54:28 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:60630
I20260925 02:54:28.191232 24968 tablet_service.cc:1514] Processing DeleteTablet for tablet e1dd0798feb94001874f561db7b266dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:54:28 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:42088
I20260925 02:54:28.191680 24832 tablet_service.cc:1514] Processing DeleteTablet for tablet e1dd0798feb94001874f561db7b266dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:54:28 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:51014
I20260925 02:54:28.192701 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 24628
I20260925 02:54:28.196087 25080 tablet_replica.cc:330] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38: stopping tablet replica
I20260925 02:54:28.196722 25080 raft_consensus.cc:2229] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:54:28.197209 25078 tablet_replica.cc:330] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355: stopping tablet replica
W20260925 02:54:28.201875 24532 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.222.193:41763 (error 108)
I20260925 02:54:28.202131 25078 raft_consensus.cc:2229] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:54:28.202344 25078 raft_consensus.cc:2258] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:54:28.205336 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:54:28.205059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:41763 (local address 127.0.0.1:60640)
0925 02:54:28.205203 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:28.205266 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.193:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260925 02:54:28.205559 24532 catalog_manager.cc:4584] TS b52d02f15dc64af3a250e743ed4f6d53 (127.12.222.193:41763): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1dd0798feb94001874f561db7b266dc: Network error: Client connection negotiation failed: client connection to 127.12.222.193:41763: connect: Connection refused (error 111)
I20260925 02:54:28.207422 25080 raft_consensus.cc:2258] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:54:28.209369 25080 ts_tablet_manager.cc:1905] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:54:28.211979 25080 ts_tablet_manager.cc:1918] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:54:28.213248 25080 log.cc:1196] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/e1dd0798feb94001874f561db7b266dc
I20260925 02:54:28.213680 25080 ts_tablet_manager.cc:1939] T e1dd0798feb94001874f561db7b266dc P 5061a83b49114b6ba65c9ea0d77f3d38: Deleting consensus metadata
I20260925 02:54:28.214354 25078 ts_tablet_manager.cc:1905] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:54:28.214699 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 24767
I20260925 02:54:28.214680 24533 catalog_manager.cc:4841] TS 5061a83b49114b6ba65c9ea0d77f3d38 (127.12.222.194:36607): tablet e1dd0798feb94001874f561db7b266dc (table test-table [id=9b521f8ce764418bb30c1fc4ce472b6f]) successfully deleted
I20260925 02:54:28.218293 25078 ts_tablet_manager.cc:1918] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:54:28.218387 25078 log.cc:1196] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/e1dd0798feb94001874f561db7b266dc
I20260925 02:54:28.220593 25078 ts_tablet_manager.cc:1939] T e1dd0798feb94001874f561db7b266dc P 6b771602d6b7413f8e004daee2b53355: Deleting consensus metadata
I20260925 02:54:28.221798 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 24904
I20260925 02:54:28.221850 24533 catalog_manager.cc:4841] TS 6b771602d6b7413f8e004daee2b53355 (127.12.222.195:38985): tablet e1dd0798feb94001874f561db7b266dc (table test-table [id=9b521f8ce764418bb30c1fc4ce472b6f]) successfully deleted
W20260925 02:54:28.229904 25050 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:54:28.229400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:38985 (local address 127.0.0.1:42104)
0925 02:54:28.229547 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 02:54:28.229561 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:54:28.229640 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:54:28.229808 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.195:38985: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 02:54:28.230492 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 24514
I20260925 02:54:28.237560 13179 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 02:54:28.571815 13179 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 02:54:28.571934 13179 mini_ranger.cc:66] Stopping Ranger...
I20260925 02:54:28.917582 13179 mini_ranger.cc:68] Stopped Ranger
2026-09-25 02:54:28.917 UTC [24096] LOG: received smart shutdown request
2026-09-25 02:54:28.921 UTC [24096] LOG: background worker "logical replication launcher" (PID 24105) exited with exit code 1
2026-09-25 02:54:28.921 UTC [24100] LOG: shutting down
2026-09-25 02:54:29.071 UTC [24096] LOG: database system is shut down
2026-09-25T02:54:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44597 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:54:29 dist-test-slave-252r krb5kdc[25090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:54:29 dist-test-slave-252r krb5kdc[25090](info): set up 2 sockets
Sep 25 02:54:29 dist-test-slave-252r krb5kdc[25090](info): commencing operation
krb5kdc: starting...
W20260925 02:54:30.159746 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:54:30 dist-test-slave-252r krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:54:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:54:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.222.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.222.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.222.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/rangeradmin_127.12.222.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.222.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/rangeradmin_127.12.222.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.222.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.222.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.222.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/rangerlookup_127.12.222.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.222.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/rangerlookup_127.12.222.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.222.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.222.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.222.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/HTTP_127.12.222.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.222.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/HTTP_127.12.222.212@KRBTEST.COM.keytab.
I20260925 02:54:30.306864 13179 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskORQp8j/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/postgres -l logfile start
2026-09-25 02:54:32.525 UTC [25120] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 02:54:32.525 UTC [25120] LOG: listening on IPv4 address "127.12.222.212", port 51271
2026-09-25 02:54:32.529 UTC [25120] LOG: listening on Unix socket "/tmp/.s.PGSQL.51271"
2026-09-25 02:54:32.562 UTC [25123] LOG: database system was shut down at 2026-09-25 02:54:31 UTC
2026-09-25 02:54:32.566 UTC [25120] LOG: database system is ready to accept connections
I20260925 02:54:33.455809 13179 mini_postgres.cc:96] Postgres bound to 51271
2026-09-25 02:54:33.461 UTC [25131] FATAL: database "slave" does not exist
127.12.222.212:51271 - accepting connections
I20260925 02:54:33.462139 13179 mini_ranger.cc:161] Starting Ranger...
I20260925 02:54:33.475771 13179 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 02:54:33.898207 13179 mini_ranger.cc:87] Created ranger Postgres database
I20260925 02:54:33.898296 13179 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-admin
2026-09-25 02:54:34,158 [I] DB FLAVOR :POSTGRES
2026-09-25 02:54:34,158 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:54:34,158 [I] Checking connection..
2026-09-25 02:54:34,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 02:54:34,397 [I] Checking connection passed.
2026-09-25 02:54:34,398 [I] --------- Verifying version history table ---------
2026-09-25 02:54:34,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:54:34,640 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:54:34,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:54:34,876 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:54:34,876 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 02:54:34,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 02:54:35.080 UTC [25195] WARNING: there is no transaction in progress
2026-09-25 02:54:35,086 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 02:54:35,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:54:35,300 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 02:54:35,300 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:54:35,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 02:54:35,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 02:54:35,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-252r.c.gcp-upstream.internal','N') ;"
2026-09-25 02:54:35,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:54:36,149 [I] Table x_portal_user does not exist in database ranger
2026-09-25 02:54:36,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:54:36,353 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 02:54:36,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:54:36,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:54:36,792 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 02:54:36,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 02:54:37.663 UTC [25304] WARNING: there is no transaction in progress
2026-09-25 02:54:37.679 UTC [25304] WARNING: there is no transaction in progress
2026-09-25 02:54:37.694 UTC [25304] WARNING: there is no transaction in progress
2026-09-25 02:54:37.708 UTC [25304] WARNING: there is no transaction in progress
2026-09-25 02:54:37.722 UTC [25304] WARNING: there is no transaction in progress
2026-09-25 02:54:37.738 UTC [25304] WARNING: there is no transaction in progress
2026-09-25 02:54:37.786 UTC [25304] WARNING: there is no transaction in progress
2026-09-25 02:54:37.792 UTC [25304] WARNING: there is no transaction in progress
2026-09-25 02:54:37.805 UTC [25304] WARNING: there is no transaction in progress
2026-09-25 02:54:37.814 UTC [25304] WARNING: there is no transaction in progress
2026-09-25 02:54:37.824 UTC [25304] WARNING: there is no transaction in progress
2026-09-25 02:54:37.835 UTC [25304] WARNING: there is no transaction in progress
2026-09-25 02:54:37.844 UTC [25304] WARNING: there is no transaction in progress
2026-09-25 02:54:37.850 UTC [25304] WARNING: there is no transaction in progress
2026-09-25 02:54:37.856 UTC [25304] WARNING: there is no transaction in progress
2026-09-25 02:54:37.863 UTC [25304] WARNING: there is no transaction in progress
2026-09-25 02:54:38,442 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 02:54:38,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 02:54:38,663 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 02:54:38,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:54:38,880 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 02:54:38,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:54:39,092 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 02:54:39,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:54:39,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:54:39,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-252r.c.gcp-upstream.internal';"
2026-09-25 02:54:39,761 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 02:54:39,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 02:54:39,976 [I] DB_PATCHES have already been applied
I20260925 02:54:39.981257 13179 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-admin:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskORQp8j/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 02:54:39.981314 13179 mini_ranger.cc:193] Using host: 127.12.222.212
I20260925 02:54:39.984633 13179 mini_ranger.cc:227] Ranger admin URL: http://127.12.222.212:60941
Sep 25, 2026 2:54:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:54:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:54:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:54:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:54:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:54:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:54:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 02:54:41.034804 13179 mini_ranger.cc:160] Time spent starting Ranger: real 7.573s user 0.002s sys 0.006s
Sep 25, 2026 2:54:59 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:54:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 2:54:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:54:59 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:55:00 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 02:55:00.832051 13179 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.222.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.222.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.222.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/rangerkms_127.12.222.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.222.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/rangerkms_127.12.222.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.222.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.222.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.222.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/HTTP_127.12.222.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.222.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/HTTP_127.12.222.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 02:55:00 dist-test-slave-252r krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304900, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 02:55:00.924419 13179 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 02:55:00.932855 13179 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 02:55:01.757591 13179 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 02:55:01.757658 13179 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-kms
I20260925 02:55:01.757671 13179 mini_ranger_kms.cc:227] Using postgres at 127.12.222.212:51271
2026-09-25 02:55:01,925 [I] DB FLAVOR :POSTGRES
2026-09-25 02:55:01,926 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:55:01,926 [I] Checking connection
2026-09-25 02:55:01,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 02:55:02,166 [I] connection success
2026-09-25 02:55:02,166 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 02:55:02,166 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 02:55:02,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 02:55:02,381 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 02:55:02,381 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:55:02,381 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 02:55:02,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.222.212:51271/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 02:55:02,620 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 02:55:03.051270 13179 mini_ranger_kms.cc:310] Created kms service
I20260925 02:55:03.255553 13179 mini_ranger_kms.cc:326] Created kudu user
I20260925 02:55:03.298236 13179 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 02:55:03 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304900, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
I20260925 02:55:03.686862 13179 mini_ranger_kms.cc:384] Added ranger policy
I20260925 02:55:03.687011 13179 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-kms:/tmp/dist-test-taskORQp8j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskORQp8j/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 02:55:03.687031 13179 mini_ranger_kms.cc:240] Using host: 127.12.222.212
I20260925 02:55:03.690667 13179 mini_ranger_kms.cc:276] Ranger KMS PID: 25484
I20260925 02:55:03.690744 13179 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.222.212:39947
Sep 25, 2026 2:55:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:55:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:55:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:55:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:55:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:55:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 02:55:05.836912 13179 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.912s user 0.011s sys 0.140s
I20260925 02:55:05.837040 13179 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 02:55:05.837118 13179 mini_ranger_kms.cc:401] 127.12.222.212:39947/kms/v1/keys
Sep 25 02:55:05 dist-test-slave-252r krb5kdc[25090](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304905, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.222.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 02:55:06 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304905, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.222.212@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
Sep 25 02:55:06 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304905, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.222.212@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
Sep 25 02:55:08 dist-test-slave-252r krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:55:08.917619 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:38225
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:38225
--ranger_config_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.222.212:39947/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:09.022711 25535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:09.022943 25535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:09.022982 25535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:09.026957 25535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:09.027004 25535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:09.027021 25535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:09.027035 25535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:09.027071 25535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:09.032094 25535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.222.212:39947/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:38225
--ranger_config_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:38225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:09.033100 25535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:09.034063 25535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:09.042486 25541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:09.042580 25540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:09.042765 25543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:09.042673 25535 server_base.cc:1034] running on GCE node
I20260925 02:55:09.043321 25535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:09.044606 25535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:09.045794 25535 hybrid_clock.cc:648] HybridClock initialized: now 1790304909045764 us; error 42 us; skew 500 ppm
Sep 25 02:55:09 dist-test-slave-252r krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304909, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:09.048916 25535 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:55:09.049763 25535 webserver.cc:460] Webserver started at http://127.12.222.254:41651/ using document root <none> and password file <none>
I20260925 02:55:09.050318 25535 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:09.050366 25535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:09.050562 25535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:55:09 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304909, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
I20260925 02:55:09.097685 25535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "eec9fdd8beac4f659ede216819aff54a"
format_stamp: "Formatted at 2026-09-25 02:55:09 on dist-test-slave-252r"
server_key: "bf97579b0188633c634d509f739f7530"
server_key_iv: "d52ef2272058f6bf048be0c13c211391"
server_key_version: "kuduclusterkey@0"
I20260925 02:55:09.098263 25535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "eec9fdd8beac4f659ede216819aff54a"
format_stamp: "Formatted at 2026-09-25 02:55:09 on dist-test-slave-252r"
server_key: "bf97579b0188633c634d509f739f7530"
server_key_iv: "d52ef2272058f6bf048be0c13c211391"
server_key_version: "kuduclusterkey@0"
I20260925 02:55:09.102133 25535 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 02:55:09.114622 25550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:09.115949 25535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:55:09.116113 25535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "eec9fdd8beac4f659ede216819aff54a"
format_stamp: "Formatted at 2026-09-25 02:55:09 on dist-test-slave-252r"
server_key: "bf97579b0188633c634d509f739f7530"
server_key_iv: "d52ef2272058f6bf048be0c13c211391"
server_key_version: "kuduclusterkey@0"
I20260925 02:55:09.116230 25535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:09.130713 25535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:09.133392 25535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:09.133922 25535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:09.140923 25535 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:38225
I20260925 02:55:09.140923 25602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:38225 every 8 connection(s)
I20260925 02:55:09.141882 25535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:09.143901 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 25535
I20260925 02:55:09.144032 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:55:09.144356 13179 external_mini_cluster.cc:1428] Setting key 95bd7db12ba2491649677ab559b55f1a
I20260925 02:55:09.144982 25603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:09.151878 25603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a: Bootstrap starting.
Sep 25 02:55:09 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:09.154395 25603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:09.155642 25603 log.cc:824] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:09.158058 25603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a: No bootstrap required, opened a new log
I20260925 02:55:09.158206 25606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:09.145669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35150 (local address 127.12.222.254:38225)
0925 02:55:09.146053 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:09.146072 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:09.146292 (+ 220us) server_negotiation.cc:407] Connection header received
0925 02:55:09.147049 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:09.147065 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:09.147360 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:09.147671 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:09.148531 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:09.149445 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:09.150287 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:09.150768 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:09.150943 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:09.153498 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:09.153503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:09.153513 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:09.153538 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:09.155137 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:09.155574 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:09.155580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:09.155592 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:09.155690 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:09.156009 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:09.156013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:09.156015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:09.156487 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:09.157199 (+ 712us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:09.157327 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 02:55:09.157521 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":127,"threads_started":1}
I20260925 02:55:09.161790 25603 raft_consensus.cc:348] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9fdd8beac4f659ede216819aff54a" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38225 } }
I20260925 02:55:09.162068 25603 raft_consensus.cc:374] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:09.162122 25603 raft_consensus.cc:729] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eec9fdd8beac4f659ede216819aff54a, State: Initialized, Role: FOLLOWER
I20260925 02:55:09.162556 25603 consensus_queue.cc:260] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9fdd8beac4f659ede216819aff54a" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38225 } }
I20260925 02:55:09.162691 25603 raft_consensus.cc:388] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:09.162746 25603 raft_consensus.cc:482] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:09.162827 25603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:09.163759 25603 raft_consensus.cc:504] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9fdd8beac4f659ede216819aff54a" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38225 } }
I20260925 02:55:09.164115 25603 leader_election.cc:302] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eec9fdd8beac4f659ede216819aff54a; no voters:
I20260925 02:55:09.164454 25603 leader_election.cc:288] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:09.164569 25608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:09.164872 25608 raft_consensus.cc:686] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [term 1 LEADER]: Becoming Leader. State: Replica: eec9fdd8beac4f659ede216819aff54a, State: Running, Role: LEADER
I20260925 02:55:09.165233 25608 consensus_queue.cc:237] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9fdd8beac4f659ede216819aff54a" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38225 } }
I20260925 02:55:09.165695 25603 sys_catalog.cc:564] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:09.166357 25608 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eec9fdd8beac4f659ede216819aff54a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9fdd8beac4f659ede216819aff54a" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38225 } } }
I20260925 02:55:09.166550 25608 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:09.166810 25608 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eec9fdd8beac4f659ede216819aff54a. Latest consensus state: current_term: 1 leader_uuid: "eec9fdd8beac4f659ede216819aff54a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9fdd8beac4f659ede216819aff54a" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38225 } } }
I20260925 02:55:09.166950 25608 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:09.167248 25617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:09.170650 25617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:09.171062 25603 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 02:55:09.180574 25617 catalog_manager.cc:1347] Generated new cluster ID: aaee6c8f3e204c65b169e606430d60c1
I20260925 02:55:09.180759 25617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:09.206547 25617 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:09.207737 25617 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:09.220264 25617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a: Generated new TSK 0
I20260925 02:55:09.221216 25617 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
I20260925 02:55:09.843005 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.222.212:39947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38225
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:09.990065 25648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:09.990312 25648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:09.990384 25648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:09.994642 25648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:09.994714 25648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:09.994805 25648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:55:09.999848 25648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.222.212:39947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:10.000815 25648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:10.001727 25648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:10.011049 25656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:10.011205 25648 server_base.cc:1034] running on GCE node
W20260925 02:55:10.011061 25654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:10.011825 25653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:10.012270 25648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:10.012924 25648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:10.014138 25648 hybrid_clock.cc:648] HybridClock initialized: now 1790304910014107 us; error 37 us; skew 500 ppm
Sep 25 02:55:10 dist-test-slave-252r krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:10.017591 25648 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:55:10.019018 25648 webserver.cc:460] Webserver started at http://127.12.222.193:45679/ using document root <none> and password file <none>
I20260925 02:55:10.019850 25648 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:10.019986 25648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:10.020279 25648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:55:10 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
I20260925 02:55:10.037025 25648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "b6df70ed89af480c8e2e4637c408d83f"
format_stamp: "Formatted at 2026-09-25 02:55:10 on dist-test-slave-252r"
server_key: "f95b87d4e3e96733862ad7520e679b1e"
server_key_iv: "1621badf309067c03d0490fb1a4a1425"
server_key_version: "kuduclusterkey@0"
I20260925 02:55:10.037817 25648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "b6df70ed89af480c8e2e4637c408d83f"
format_stamp: "Formatted at 2026-09-25 02:55:10 on dist-test-slave-252r"
server_key: "f95b87d4e3e96733862ad7520e679b1e"
server_key_iv: "1621badf309067c03d0490fb1a4a1425"
server_key_version: "kuduclusterkey@0"
I20260925 02:55:10.045353 25648 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.000s
I20260925 02:55:10.060806 25665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:10.062363 25648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:55:10.062544 25648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "b6df70ed89af480c8e2e4637c408d83f"
format_stamp: "Formatted at 2026-09-25 02:55:10 on dist-test-slave-252r"
server_key: "f95b87d4e3e96733862ad7520e679b1e"
server_key_iv: "1621badf309067c03d0490fb1a4a1425"
server_key_version: "kuduclusterkey@0"
I20260925 02:55:10.062729 25648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:10.081022 25648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:10.085283 25648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:10.085531 25648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:10.086229 25648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:10.087229 25648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:10.087337 25648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:10.087473 25648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:10.087549 25648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:10.097528 25648 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:34263
I20260925 02:55:10.098544 25648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:55:10.100759 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 25648
I20260925 02:55:10.099260 25778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:34263 every 8 connection(s)
I20260925 02:55:10.100901 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:55:10.101228 13179 external_mini_cluster.cc:1428] Setting key d371adfec9c34d19ac00fd78244db134
Sep 25 02:55:10 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.222.194@KRBTEST.COM; defaulting to no policy
I20260925 02:55:10.119415 25781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:10.102296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38225 (local address 127.12.222.193:40531)
0925 02:55:10.104844 (+ 2548us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:10.104900 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:10.106838 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:10.107541 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:10.107550 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:10.107853 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:10.108464 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:10.108483 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:10.109600 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:10.109604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:10.110954 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:10.110969 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:10.111765 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:10.111769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:10.111968 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:10.112576 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:10.112592 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:10.113881 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:10.116247 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:10.116254 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:10.116268 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:10.116606 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:10.116943 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:10.116949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:10.116954 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:10.117134 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:10.117714 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:10.117720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:10.118039 (+ 319us) client_negotiation.cc:769] Sending connection context
0925 02:55:10.118242 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 02:55:10.118526 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2351,"thread_start_us":131,"threads_started":1}
I20260925 02:55:10.120146 25782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:10.103493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:40531 (local address 127.12.222.254:38225)
0925 02:55:10.106404 (+ 2911us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:10.106421 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:10.106436 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:55:10.107135 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:10.107138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:10.107186 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:10.107267 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:10.108626 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:10.109457 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:10.111099 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:10.111468 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:10.111586 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:10.114081 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:10.114087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:10.114091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:10.114122 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:10.116088 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:10.116730 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:10.116733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:10.116735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:10.116827 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:10.117258 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:10.117263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:10.117264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:10.117557 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:10.117654 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:10.119841 (+ 2187us) server_negotiation.cc:299] Negotiation successful
0925 02:55:10.119975 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2812,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.194@KRBTEST.COM" created.
I20260925 02:55:10.121232 25779 heartbeater.cc:347] Connected to a master server at 127.12.222.254:38225
I20260925 02:55:10.121454 25779 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:10.121892 25779 heartbeater.cc:511] Master 127.12.222.254:38225 requested a full tablet report, sending...
I20260925 02:55:10.123448 25567 ts_manager.cc:194] Registered new tserver with Master: b6df70ed89af480c8e2e4637c408d83f (127.12.222.193:34263)
I20260925 02:55:10.124828 25567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:40531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
I20260925 02:55:10.144353 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.222.212:39947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38225
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:55:10 dist-test-slave-252r krb5kdc[25090](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 02:55:10.290170 25787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:10.290552 25787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:10.290704 25787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:10.296622 25787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:10.296695 25787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:10.296772 25787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:55:10.304344 25787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.222.212:39947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:10.305627 25787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:10.306964 25787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:10.321062 25793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:10.321185 25796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:10.321074 25794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:10.323645 25787 server_base.cc:1034] running on GCE node
I20260925 02:55:10.324039 25787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:10.324759 25787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:10.325984 25787 hybrid_clock.cc:648] HybridClock initialized: now 1790304910325964 us; error 35 us; skew 500 ppm
Sep 25 02:55:10 dist-test-slave-252r krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:10.329267 25787 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:55:10.330186 25787 webserver.cc:460] Webserver started at http://127.12.222.194:38307/ using document root <none> and password file <none>
I20260925 02:55:10.330919 25787 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:10.331037 25787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:10.331296 25787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:55:10 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
I20260925 02:55:10.349272 25787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "e349bfeacbfe4a01aa7f2096da322092"
format_stamp: "Formatted at 2026-09-25 02:55:10 on dist-test-slave-252r"
server_key: "9d384709a586534a491ea8203dc184de"
server_key_iv: "8e8c9e068790ccd853cb7eb3ca5610a6"
server_key_version: "kuduclusterkey@0"
I20260925 02:55:10.349973 25787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "e349bfeacbfe4a01aa7f2096da322092"
format_stamp: "Formatted at 2026-09-25 02:55:10 on dist-test-slave-252r"
server_key: "9d384709a586534a491ea8203dc184de"
server_key_iv: "8e8c9e068790ccd853cb7eb3ca5610a6"
server_key_version: "kuduclusterkey@0"
I20260925 02:55:10.354768 25787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:55:10.374132 25803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:10.375869 25787 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 02:55:10.376027 25787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "e349bfeacbfe4a01aa7f2096da322092"
format_stamp: "Formatted at 2026-09-25 02:55:10 on dist-test-slave-252r"
server_key: "9d384709a586534a491ea8203dc184de"
server_key_iv: "8e8c9e068790ccd853cb7eb3ca5610a6"
server_key_version: "kuduclusterkey@0"
I20260925 02:55:10.376201 25787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:10.409359 25787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:10.412626 25787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:10.412809 25787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:10.413403 25787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:10.414249 25787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:10.414317 25787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:10.414399 25787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:10.414443 25787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:10.425022 25787 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:36629
I20260925 02:55:10.425043 25916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:36629 every 8 connection(s)
I20260925 02:55:10.426168 25787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:55:10.426453 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 25787
I20260925 02:55:10.426591 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:55:10.426903 13179 external_mini_cluster.cc:1428] Setting key b7126d238fac79606334820a17ebaef4
Sep 25 02:55:10 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.222.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.195@KRBTEST.COM" created.
I20260925 02:55:10.441740 25782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:10.428193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:37135 (local address 127.12.222.254:38225)
0925 02:55:10.428417 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:10.428422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:10.428956 (+ 534us) server_negotiation.cc:407] Connection header received
0925 02:55:10.429909 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:10.429920 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:10.429974 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:10.430069 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:10.431484 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:10.432317 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:10.433972 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:10.434299 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:10.434403 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:10.437255 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:10.437259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:10.437261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:10.437289 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:10.439112 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:10.439620 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:10.439625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:10.439628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:10.439732 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:10.440116 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:10.440121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:10.440124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:10.440421 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:10.440530 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:10.441334 (+ 804us) server_negotiation.cc:299] Negotiation successful
0925 02:55:10.441485 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260925 02:55:10.442785 25919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:10.428404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38225 (local address 127.12.222.194:37135)
0925 02:55:10.428825 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:10.428871 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:10.429698 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:10.430238 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:10.430245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:10.430639 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:10.431302 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:10.431323 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:10.432476 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:10.432482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:10.433842 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:10.433856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:10.434472 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:10.434478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:10.434695 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:10.435336 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:10.435358 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:10.437051 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:10.439227 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:10.439231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:10.439240 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:10.439511 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:10.439834 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:10.439836 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:10.439837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:10.440020 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:10.440597 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:10.440605 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:10.440989 (+ 384us) client_negotiation.cc:769] Sending connection context
0925 02:55:10.441260 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 02:55:10.441624 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":126,"threads_started":1}
I20260925 02:55:10.444281 25917 heartbeater.cc:347] Connected to a master server at 127.12.222.254:38225
I20260925 02:55:10.444537 25917 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:10.445134 25917 heartbeater.cc:511] Master 127.12.222.254:38225 requested a full tablet report, sending...
I20260925 02:55:10.446389 25567 ts_manager.cc:194] Registered new tserver with Master: e349bfeacbfe4a01aa7f2096da322092 (127.12.222.194:36629)
I20260925 02:55:10.447031 25567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:37135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
I20260925 02:55:10.465157 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.222.212:39947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38225
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:55:10 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
W20260925 02:55:10.611486 25922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:10.611785 25922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:10.611838 25922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:10.617194 25922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:10.617246 25922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:10.617317 25922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:55:10.624145 25922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.222.212:39947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:10.625322 25922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:10.626570 25922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:10.639400 25928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:10.639398 25930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:10.640340 25927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:10.640805 25922 server_base.cc:1034] running on GCE node
I20260925 02:55:10.641237 25922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:10.641902 25922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:10.645175 25922 hybrid_clock.cc:648] HybridClock initialized: now 1790304910645107 us; error 74 us; skew 500 ppm
Sep 25 02:55:10 dist-test-slave-252r krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:10.648392 25922 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:55:10.649259 25922 webserver.cc:460] Webserver started at http://127.12.222.195:42963/ using document root <none> and password file <none>
I20260925 02:55:10.649840 25922 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:10.649899 25922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:10.650084 25922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:55:10 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
I20260925 02:55:10.676465 25922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "68f2c0418f0d4179b8160b0eded6988f"
format_stamp: "Formatted at 2026-09-25 02:55:10 on dist-test-slave-252r"
server_key: "07f4a70d4a73d1c13c82e4b82401248c"
server_key_iv: "a118e3d37f2348226bb347d304642ae7"
server_key_version: "kuduclusterkey@0"
I20260925 02:55:10.677147 25922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "68f2c0418f0d4179b8160b0eded6988f"
format_stamp: "Formatted at 2026-09-25 02:55:10 on dist-test-slave-252r"
server_key: "07f4a70d4a73d1c13c82e4b82401248c"
server_key_iv: "a118e3d37f2348226bb347d304642ae7"
server_key_version: "kuduclusterkey@0"
I20260925 02:55:10.681891 25922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:55:10.700590 25937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:10.702104 25922 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260925 02:55:10.702211 25922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "68f2c0418f0d4179b8160b0eded6988f"
format_stamp: "Formatted at 2026-09-25 02:55:10 on dist-test-slave-252r"
server_key: "07f4a70d4a73d1c13c82e4b82401248c"
server_key_iv: "a118e3d37f2348226bb347d304642ae7"
server_key_version: "kuduclusterkey@0"
I20260925 02:55:10.702322 25922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:10.722704 25922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:10.725876 25922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:10.726047 25922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:10.726625 25922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:10.727555 25922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:10.727612 25922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:10.727659 25922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:10.727687 25922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:10.737246 25922 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:45507
I20260925 02:55:10.737397 26050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:45507 every 8 connection(s)
I20260925 02:55:10.738238 25922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:55:10.742722 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 25922
I20260925 02:55:10.742852 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:55:10.743175 13179 external_mini_cluster.cc:1428] Setting key 2dde8d276059fbeb16a8ce920e2b0ea6
Sep 25 02:55:10 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:10.754869 25782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:10.741713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:56371 (local address 127.12.222.254:38225)
0925 02:55:10.741819 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:10.741824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:10.741840 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:55:10.742505 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:10.742509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:10.742554 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:10.742625 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:10.745003 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:10.745810 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:10.747435 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:10.747777 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:10.747880 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:10.750801 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:10.750805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:10.750808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:10.750835 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:10.752561 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:10.753167 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:10.753171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:10.753174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:10.753270 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:10.753684 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:10.753688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:10.753690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:10.753922 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:10.754030 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:10.754573 (+ 543us) server_negotiation.cc:299] Negotiation successful
0925 02:55:10.754696 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:55:10.755679 26053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:10.740670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38225 (local address 127.12.222.195:56371)
0925 02:55:10.741227 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:10.741283 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:10.742289 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:10.743769 (+ 1480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:10.743777 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:10.744135 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:10.744790 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:10.744809 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:10.745982 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:10.745985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:10.747308 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:10.747319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:10.747998 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:10.748003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:10.748194 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:10.748853 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:10.748873 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:10.750602 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:10.752678 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:10.752685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:10.752698 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:10.753031 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:10.753372 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:10.753375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:10.753377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:10.753541 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:10.754032 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:10.754037 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:10.754294 (+ 257us) client_negotiation.cc:769] Sending connection context
0925 02:55:10.754472 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 02:55:10.754740 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":149,"threads_started":1}
I20260925 02:55:10.756944 26051 heartbeater.cc:347] Connected to a master server at 127.12.222.254:38225
I20260925 02:55:10.757159 26051 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:10.757625 26051 heartbeater.cc:511] Master 127.12.222.254:38225 requested a full tablet report, sending...
I20260925 02:55:10.758682 25567 ts_manager.cc:194] Registered new tserver with Master: 68f2c0418f0d4179b8160b0eded6988f (127.12.222.195:45507)
I20260925 02:55:10.759370 25567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:56371
I20260925 02:55:10.768872 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:55:10.769584 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 25648
I20260925 02:55:10.776542 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 25787
I20260925 02:55:10.782971 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 25922
I20260925 02:55:10.791368 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 25535
I20260925 02:55:10.799093 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:38225
--webserver_interface=127.12.222.254
--webserver_port=41651
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:38225
--ranger_config_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.222.212:39947/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.222.212:39947/kms with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:10.901968 26054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:10.902169 26054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:10.902204 26054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:10.906032 26054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:10.906080 26054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:10.906098 26054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:10.906157 26054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:10.906184 26054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:10.911091 26054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.222.212:39947/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:38225
--ranger_config_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:38225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=41651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:10.912003 26054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:10.912878 26054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:10.920246 26059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:10.920331 26060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:10.920379 26054 server_base.cc:1034] running on GCE node
W20260925 02:55:10.920433 26062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:10.920984 26054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:10.921919 26054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:10.923098 26054 hybrid_clock.cc:648] HybridClock initialized: now 1790304910923073 us; error 36 us; skew 500 ppm
Sep 25 02:55:10 dist-test-slave-252r krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:10.925711 26054 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:55:10.926483 26054 webserver.cc:460] Webserver started at http://127.12.222.254:41651/ using document root <none> and password file <none>
I20260925 02:55:10.926980 26054 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:10.927073 26054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:55:10 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304910, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
I20260925 02:55:10.987370 26054 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:55:10.989511 26070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:10.990726 26054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:55:10.990877 26054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "eec9fdd8beac4f659ede216819aff54a"
format_stamp: "Formatted at 2026-09-25 02:55:09 on dist-test-slave-252r"
server_key: "bf97579b0188633c634d509f739f7530"
server_key_iv: "d52ef2272058f6bf048be0c13c211391"
server_key_version: "kuduclusterkey@0"
I20260925 02:55:10.991400 26054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:11.008016 26054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:11.010314 26054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:11.010504 26054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:11.017305 26054 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:38225
I20260925 02:55:11.017338 26122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:38225 every 8 connection(s)
I20260925 02:55:11.018354 26054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:11.022923 26123 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:55:11.024525 26123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a: Bootstrap starting.
I20260925 02:55:11.025597 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 26054
I20260925 02:55:11.026016 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.222.212:39947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:34263
--local_ip_for_outbound_sockets=127.12.222.193
--tserver_master_addrs=127.12.222.254:38225
--webserver_port=45679
--webserver_interface=127.12.222.193
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.222.212:39947/kms with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:55:11.039832 26123 log.cc:824] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:11.045749 26123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:55:11.046096 26123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a: Bootstrap complete.
I20260925 02:55:11.049736 26123 raft_consensus.cc:348] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9fdd8beac4f659ede216819aff54a" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38225 } }
I20260925 02:55:11.050102 26123 raft_consensus.cc:729] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eec9fdd8beac4f659ede216819aff54a, State: Initialized, Role: FOLLOWER
I20260925 02:55:11.050599 26123 consensus_queue.cc:260] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9fdd8beac4f659ede216819aff54a" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38225 } }
I20260925 02:55:11.050735 26123 raft_consensus.cc:388] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:11.050832 26123 raft_consensus.cc:482] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:11.050944 26123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:55:11.052718 26123 raft_consensus.cc:504] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9fdd8beac4f659ede216819aff54a" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38225 } }
I20260925 02:55:11.053025 26123 leader_election.cc:302] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eec9fdd8beac4f659ede216819aff54a; no voters:
I20260925 02:55:11.053318 26123 leader_election.cc:288] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:55:11.053391 26127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:55:11.053606 26127 raft_consensus.cc:686] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [term 2 LEADER]: Becoming Leader. State: Replica: eec9fdd8beac4f659ede216819aff54a, State: Running, Role: LEADER
I20260925 02:55:11.053871 26127 consensus_queue.cc:237] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9fdd8beac4f659ede216819aff54a" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38225 } }
I20260925 02:55:11.054436 26123 sys_catalog.cc:564] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:11.054735 26127 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eec9fdd8beac4f659ede216819aff54a. Latest consensus state: current_term: 2 leader_uuid: "eec9fdd8beac4f659ede216819aff54a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9fdd8beac4f659ede216819aff54a" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38225 } } }
I20260925 02:55:11.054921 26127 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:11.054715 26128 sys_catalog.cc:455] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eec9fdd8beac4f659ede216819aff54a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9fdd8beac4f659ede216819aff54a" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38225 } } }
I20260925 02:55:11.055279 26128 sys_catalog.cc:458] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:11.056272 26131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:11.056639 26123 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 02:55:11.062317 26131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:11.063256 26131 catalog_manager.cc:1259] Loaded cluster ID: aaee6c8f3e204c65b169e606430d60c1
I20260925 02:55:11.063330 26131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:11.066170 26131 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:11.066864 26131 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a: Loaded TSK: 0
I20260925 02:55:11.068120 26131 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:55:11.157567 26125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:11.157832 26125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:11.157886 26125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:11.163297 26125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:11.163357 26125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:11.163436 26125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:55:11.170032 26125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.222.212:39947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:34263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=45679
--tserver_master_addrs=127.12.222.254:38225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:11.170992 26125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:11.172040 26125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:11.183173 26163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:11.183560 26160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:11.184306 26161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:11.184585 26125 server_base.cc:1034] running on GCE node
I20260925 02:55:11.185022 26125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:11.185874 26125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:11.187083 26125 hybrid_clock.cc:648] HybridClock initialized: now 1790304911187060 us; error 36 us; skew 500 ppm
Sep 25 02:55:11 dist-test-slave-252r krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:11.190313 26125 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:55:11.191299 26125 webserver.cc:460] Webserver started at http://127.12.222.193:45679/ using document root <none> and password file <none>
I20260925 02:55:11.191928 26125 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:11.192028 26125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:55:11 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
I20260925 02:55:11.217510 26125 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260925 02:55:11.222344 26171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:11.223820 26125 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260925 02:55:11.223929 26125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "b6df70ed89af480c8e2e4637c408d83f"
format_stamp: "Formatted at 2026-09-25 02:55:10 on dist-test-slave-252r"
server_key: "f95b87d4e3e96733862ad7520e679b1e"
server_key_iv: "1621badf309067c03d0490fb1a4a1425"
server_key_version: "kuduclusterkey@0"
I20260925 02:55:11.224612 26125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:11.243732 26125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:11.246979 26125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:11.247159 26125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:11.247699 26125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:11.248545 26125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:11.248636 26125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:11.248739 26125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:11.248800 26125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:11.258546 26125 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:34263
I20260925 02:55:11.258704 26284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:34263 every 8 connection(s)
I20260925 02:55:11.259479 26125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:55:11.267323 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 26125
I20260925 02:55:11.267815 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.222.212:39947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:36629
--local_ip_for_outbound_sockets=127.12.222.194
--tserver_master_addrs=127.12.222.254:38225
--webserver_port=38307
--webserver_interface=127.12.222.194
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.222.212:39947/kms with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:55:11 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:11.286609 26287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:11.263558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:33549 (local address 127.12.222.254:38225)
0925 02:55:11.263930 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:11.263952 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:11.264758 (+ 806us) server_negotiation.cc:407] Connection header received
0925 02:55:11.266007 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:11.266018 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:11.266305 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:11.266638 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:11.267808 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.269234 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.270304 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.270720 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.270892 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:11.273860 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:11.273866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:11.273883 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:11.273915 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:11.276383 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:11.276992 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:11.276997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:11.277002 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:11.277133 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:11.277550 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:11.277553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:11.277555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:11.277975 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:11.278102 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:11.285323 (+ 7221us) server_negotiation.cc:299] Negotiation successful
0925 02:55:11.285600 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":109,"threads_started":1}
I20260925 02:55:11.287833 26288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:11.263774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38225 (local address 127.12.222.193:33549)
0925 02:55:11.264567 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:11.264638 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:11.265603 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:11.266717 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:11.266741 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:11.267027 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:11.267614 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:11.267635 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.269411 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.269413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:11.270195 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:11.270202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.270868 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.270872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:11.271026 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:11.271535 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:11.271548 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:11.273645 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:11.276520 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:11.276527 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:11.276541 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:11.276856 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:11.277243 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:11.277246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:11.277248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:11.277419 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:11.278157 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:11.278163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:11.280669 (+ 2506us) client_negotiation.cc:769] Sending connection context
0925 02:55:11.282990 (+ 2321us) client_negotiation.cc:240] Negotiation successful
0925 02:55:11.286979 (+ 3989us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":147,"threads_started":1}
I20260925 02:55:11.290992 26285 heartbeater.cc:347] Connected to a master server at 127.12.222.254:38225
I20260925 02:55:11.291213 26285 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:11.291767 26285 heartbeater.cc:511] Master 127.12.222.254:38225 requested a full tablet report, sending...
W20260925 02:55:11.293891 26285 heartbeater.cc:644] Failed to heartbeat to 127.12.222.254:38225 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:55:11.294003 26285 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:11.294154 26285 heartbeater.cc:511] Master 127.12.222.254:38225 requested a full tablet report, sending...
I20260925 02:55:11.295150 26285 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:11.295305 26285 heartbeater.cc:511] Master 127.12.222.254:38225 requested a full tablet report, sending...
W20260925 02:55:11.296162 26285 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:55:11.392354 26289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:11.392554 26289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:11.392649 26289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:11.396667 26289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:11.396716 26289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:11.396786 26289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:55:11.401584 26289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.222.212:39947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:36629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=38307
--tserver_master_addrs=127.12.222.254:38225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:11.402465 26289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:11.403388 26289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:11.415669 26295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:11.415668 26294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:11.416165 26297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:11.416358 26289 server_base.cc:1034] running on GCE node
I20260925 02:55:11.416744 26289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:11.417357 26289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:11.418536 26289 hybrid_clock.cc:648] HybridClock initialized: now 1790304911418507 us; error 46 us; skew 500 ppm
Sep 25 02:55:11 dist-test-slave-252r krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:11.421278 26289 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:55:11.422116 26289 webserver.cc:460] Webserver started at http://127.12.222.194:38307/ using document root <none> and password file <none>
I20260925 02:55:11.422825 26289 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:11.422936 26289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:55:11 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
I20260925 02:55:11.442924 26289 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:55:11.446693 26305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:11.448081 26289 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:55:11.448210 26289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "e349bfeacbfe4a01aa7f2096da322092"
format_stamp: "Formatted at 2026-09-25 02:55:10 on dist-test-slave-252r"
server_key: "9d384709a586534a491ea8203dc184de"
server_key_iv: "8e8c9e068790ccd853cb7eb3ca5610a6"
server_key_version: "kuduclusterkey@0"
I20260925 02:55:11.448755 26289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:11.477101 26289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:11.479773 26289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:11.479952 26289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:11.480600 26289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:11.481528 26289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:11.481595 26289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:11.481655 26289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:11.481688 26289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:11.491915 26289 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:36629
I20260925 02:55:11.492089 26418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:36629 every 8 connection(s)
I20260925 02:55:11.493002 26289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:55:11.500141 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 26289
I20260925 02:55:11.500629 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.222.212:39947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:45507
--local_ip_for_outbound_sockets=127.12.222.195
--tserver_master_addrs=127.12.222.254:38225
--webserver_port=42963
--webserver_interface=127.12.222.195
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.222.212:39947/kms with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:55:11 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:11.509764 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:11.495274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38225 (local address 127.12.222.194:48671)
0925 02:55:11.495767 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:11.495816 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:11.496766 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:11.497308 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:11.497315 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:11.497556 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:11.497995 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:11.498008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.498983 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.498986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:11.500288 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:11.500299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.501131 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.501138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:11.501352 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:11.501993 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:11.502018 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:11.503602 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:11.505729 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:11.505734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:11.505748 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:11.506081 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:11.506430 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:11.506433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:11.506435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:11.506617 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:11.507928 (+ 1311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:11.507934 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:11.508242 (+ 308us) client_negotiation.cc:769] Sending connection context
0925 02:55:11.508478 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 02:55:11.508779 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":141,"threads_started":1}
I20260925 02:55:11.510444 26287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:11.494830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:48671 (local address 127.12.222.254:38225)
0925 02:55:11.494985 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:11.494990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:11.495907 (+ 917us) server_negotiation.cc:407] Connection header received
0925 02:55:11.496997 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:11.497001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:11.497062 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:11.497140 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:11.498145 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.498802 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.500423 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.500751 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.500956 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:11.503788 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:11.503791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:11.503794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:11.503820 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:11.505595 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:11.506202 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:11.506207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:11.506208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:11.506307 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:11.506737 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:11.506741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:11.506742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:11.506946 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:11.507048 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:11.510115 (+ 3067us) server_negotiation.cc:299] Negotiation successful
0925 02:55:11.510270 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:55:11.511868 26419 heartbeater.cc:347] Connected to a master server at 127.12.222.254:38225
I20260925 02:55:11.512085 26419 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:11.512625 26419 heartbeater.cc:511] Master 127.12.222.254:38225 requested a full tablet report, sending...
W20260925 02:55:11.514300 26419 heartbeater.cc:644] Failed to heartbeat to 127.12.222.254:38225 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:55:11.514429 26419 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:11.514611 26419 heartbeater.cc:511] Master 127.12.222.254:38225 requested a full tablet report, sending...
I20260925 02:55:11.515463 26419 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:11.515625 26419 heartbeater.cc:511] Master 127.12.222.254:38225 requested a full tablet report, sending...
W20260925 02:55:11.516485 26419 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:55:11.642144 26423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:11.642370 26423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:11.642434 26423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:11.646312 26423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:11.646396 26423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:11.646479 26423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:55:11.651310 26423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.222.212:39947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:45507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=42963
--tserver_master_addrs=127.12.222.254:38225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:11.652323 26423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:11.653329 26423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:11.663668 26428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:11.663676 26429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:11.663919 26431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:11.669410 26423 server_base.cc:1034] running on GCE node
I20260925 02:55:11.669835 26423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:11.670527 26423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:11.671726 26423 hybrid_clock.cc:648] HybridClock initialized: now 1790304911671695 us; error 36 us; skew 500 ppm
Sep 25 02:55:11 dist-test-slave-252r krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:11.675287 26423 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:55:11.676465 26423 webserver.cc:460] Webserver started at http://127.12.222.195:42963/ using document root <none> and password file <none>
I20260925 02:55:11.677114 26423 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:11.677234 26423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:55:11 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
I20260925 02:55:11.699630 26423 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:55:11.702551 26439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:11.703867 26423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:55:11.703989 26423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "68f2c0418f0d4179b8160b0eded6988f"
format_stamp: "Formatted at 2026-09-25 02:55:10 on dist-test-slave-252r"
server_key: "07f4a70d4a73d1c13c82e4b82401248c"
server_key_iv: "a118e3d37f2348226bb347d304642ae7"
server_key_version: "kuduclusterkey@0"
I20260925 02:55:11.704620 26423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:11.748903 26423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:11.751720 26423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:11.751917 26423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:11.752589 26423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:11.753472 26423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:11.753544 26423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:11.753602 26423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:11.753631 26423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:11.763695 26423 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:45507
I20260925 02:55:11.763783 26559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:45507 every 8 connection(s)
I20260925 02:55:11.764678 26423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:55:11 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:11.773892 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 26423
I20260925 02:55:11.778462 26287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:11.766495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:47629 (local address 127.12.222.254:38225)
0925 02:55:11.766628 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:11.766644 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:11.767477 (+ 833us) server_negotiation.cc:407] Connection header received
0925 02:55:11.768443 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:11.768446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:11.768498 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:11.768595 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:11.769536 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.769996 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.771071 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.771277 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.771387 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:11.773873 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:11.773877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:11.773882 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:11.773914 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:11.776165 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:11.776823 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:11.776827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:11.776828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:11.776906 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:11.777247 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:11.777250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:11.777252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:11.777439 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:11.777504 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:11.778164 (+ 660us) server_negotiation.cc:299] Negotiation successful
0925 02:55:11.778284 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:55:11.779932 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:11.766913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38225 (local address 127.12.222.195:47629)
0925 02:55:11.767339 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:11.767396 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:11.768215 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:11.768761 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:11.768767 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:11.769006 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:11.769407 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:11.769419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.770108 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.770111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:11.770952 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:11.770960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.771425 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.771429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:11.771589 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:11.772098 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:11.772112 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:11.773634 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:11.776374 (+ 2740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:11.776380 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:11.776392 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:11.776706 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:11.777005 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:11.777008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:11.777010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:11.777157 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:11.777579 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:11.777585 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:11.777906 (+ 321us) client_negotiation.cc:769] Sending connection context
0925 02:55:11.778641 (+ 735us) client_negotiation.cc:240] Negotiation successful
0925 02:55:11.778937 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":130,"threads_started":1}
I20260925 02:55:11.781240 26560 heartbeater.cc:347] Connected to a master server at 127.12.222.254:38225
I20260925 02:55:11.781492 26560 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:11.781955 26560 heartbeater.cc:511] Master 127.12.222.254:38225 requested a full tablet report, sending...
I20260925 02:55:11.783289 26087 ts_manager.cc:194] Registered new tserver with Master: 68f2c0418f0d4179b8160b0eded6988f (127.12.222.195:45507)
I20260925 02:55:11.784811 26087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:47629
I20260925 02:55:11.785684 26564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:11.774569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35156 (local address 127.12.222.254:38225)
0925 02:55:11.775248 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:11.775256 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:11.775266 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:55:11.775295 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:11.775297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:11.775370 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:11.775475 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:11.776529 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.777246 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.778413 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:11.778703 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:11.778786 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:11.780379 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:11.780382 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:11.780384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:11.780406 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:11.783564 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:11.784050 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:11.784054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:11.784055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:11.784143 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:11.784447 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:11.784450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:11.784451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:11.784620 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:11.784756 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:11.785436 (+ 680us) server_negotiation.cc:299] Negotiation successful
0925 02:55:11.785544 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":113,"threads_started":1}
Sep 25 02:55:12 dist-test-slave-252r krb5kdc[25090](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304912, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:12.296341 26285 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:12.296648 26285 heartbeater.cc:511] Master 127.12.222.254:38225 requested a full tablet report, sending...
I20260925 02:55:12.297359 26087 ts_manager.cc:194] Registered new tserver with Master: b6df70ed89af480c8e2e4637c408d83f (127.12.222.193:34263)
I20260925 02:55:12.298178 26087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:33549
Sep 25 02:55:12 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304912, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
I20260925 02:55:12.516692 26419 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:12.517009 26419 heartbeater.cc:511] Master 127.12.222.254:38225 requested a full tablet report, sending...
I20260925 02:55:12.517776 26087 ts_manager.cc:194] Registered new tserver with Master: e349bfeacbfe4a01aa7f2096da322092 (127.12.222.194:36629)
I20260925 02:55:12.518600 26087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:48671
I20260925 02:55:12.523136 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:55:12.537547 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.524953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35170 (local address 127.12.222.254:38225)
0925 02:55:12.528426 (+ 3473us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:12.528430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:12.528444 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:55:12.528480 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:12.528483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:12.528523 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:12.528606 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:12.529566 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.530454 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.531617 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.531939 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.532033 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.533734 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:12.533738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:12.533741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:12.533766 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:12.535647 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:12.536160 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:12.536164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:12.536165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:12.536282 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:12.536627 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:12.536631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:12.536632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:12.536862 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:12.537002 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:12.537261 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 02:55:12.537375 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3394,"thread_start_us":73,"threads_started":1}
I20260925 02:55:12.541445 26087 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:55:12.545692 26087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:55:12.563978 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.556104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:45507 (local address 127.0.0.1:52038)
0925 02:55:12.557923 (+ 1819us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:12.557969 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:12.558078 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:12.559788 (+ 1710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:12.559791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:12.559806 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:12.559884 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.559891 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.561169 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.561172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.562225 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.562236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.563503 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.563506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.563611 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.563637 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:55:12.563734 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 02:55:12.563839 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1638,"mutex_wait_us":1,"thread_start_us":135,"threads_started":1}
I20260925 02:55:12.564554 26580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.556888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:36629 (local address 127.0.0.1:52188)
0925 02:55:12.558251 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:12.558263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:12.558337 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:12.558791 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:12.558796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:12.558830 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:12.558939 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.558945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.560745 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.560747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.561931 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.561941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.563460 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.563463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.563562 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.564298 (+ 736us) client_negotiation.cc:769] Sending connection context
0925 02:55:12.564378 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 02:55:12.564439 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1310,"mutex_wait_us":694,"thread_start_us":105,"threads_started":1}
I20260925 02:55:12.565042 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.557342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52188 (local address 127.12.222.194:36629)
0925 02:55:12.558322 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:12.558328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:12.558362 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:55:12.558437 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:12.558443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:12.558627 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:12.558762 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:12.559066 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.560639 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.562058 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.562741 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.563606 (+ 865us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.563718 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:12.564827 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0925 02:55:12.564921 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":93,"threads_started":1}
I20260925 02:55:12.567911 26340 tablet_service.cc:1467] Processing CreateTablet for tablet 65d4fadb2df94ea385f25139915d33f3 (DEFAULT_TABLE table=test-table [id=f86836ea86b1462bbf056bdefe7b9207]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:12.569059 26340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d4fadb2df94ea385f25139915d33f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:12.569886 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.557099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:34263 (local address 127.0.0.1:52626)
0925 02:55:12.558974 (+ 1875us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:12.558987 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:12.559059 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:12.565352 (+ 6293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:12.565355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:12.565373 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:12.565459 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.565464 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.567489 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.567492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.568571 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.568582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.569595 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.569598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.569672 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.569692 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 02:55:12.569731 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 02:55:12.569783 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1818,"thread_start_us":79,"threads_started":1}
I20260925 02:55:12.570724 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.557240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52626 (local address 127.12.222.193:34263)
0925 02:55:12.564433 (+ 7193us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:12.564439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:12.564459 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:55:12.564523 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:12.564527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:12.565069 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:12.565596 (+ 527us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:12.565714 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.567375 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.568704 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.569473 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.570283 (+ 810us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.570432 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:12.570479 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 02:55:12.570601 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7095,"thread_start_us":56,"threads_started":1}
I20260925 02:55:12.573195 26215 tablet_service.cc:1467] Processing CreateTablet for tablet 65d4fadb2df94ea385f25139915d33f3 (DEFAULT_TABLE table=test-table [id=f86836ea86b1462bbf056bdefe7b9207]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:12.574051 26215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d4fadb2df94ea385f25139915d33f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:12.576612 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.556259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52038 (local address 127.12.222.195:45507)
0925 02:55:12.559280 (+ 3021us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:12.559287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:12.559305 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:55:12.559380 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:12.559384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:12.559542 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:12.559664 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:12.560000 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.561036 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.562374 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.563157 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.563276 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.563435 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:12.576338 (+ 12903us) server_negotiation.cc:299] Negotiation successful
0925 02:55:12.576450 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2925,"thread_start_us":130,"threads_started":1}
I20260925 02:55:12.579191 26586 tablet_bootstrap.cc:492] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092: Bootstrap starting.
I20260925 02:55:12.579071 26494 tablet_service.cc:1467] Processing CreateTablet for tablet 65d4fadb2df94ea385f25139915d33f3 (DEFAULT_TABLE table=test-table [id=f86836ea86b1462bbf056bdefe7b9207]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:12.579955 26494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d4fadb2df94ea385f25139915d33f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:12.581431 26586 tablet_bootstrap.cc:654] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:12.582238 26586 log.cc:824] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:12.584136 26586 tablet_bootstrap.cc:492] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092: No bootstrap required, opened a new log
I20260925 02:55:12.584334 26586 ts_tablet_manager.cc:1397] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 02:55:12.588328 26586 raft_consensus.cc:348] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f2c0418f0d4179b8160b0eded6988f" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45507 } } peers { permanent_uuid: "e349bfeacbfe4a01aa7f2096da322092" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36629 } } peers { permanent_uuid: "b6df70ed89af480c8e2e4637c408d83f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 34263 } }
I20260925 02:55:12.588601 26586 raft_consensus.cc:374] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:12.588656 26586 raft_consensus.cc:729] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e349bfeacbfe4a01aa7f2096da322092, State: Initialized, Role: FOLLOWER
I20260925 02:55:12.589090 26586 consensus_queue.cc:260] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f2c0418f0d4179b8160b0eded6988f" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45507 } } peers { permanent_uuid: "e349bfeacbfe4a01aa7f2096da322092" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36629 } } peers { permanent_uuid: "b6df70ed89af480c8e2e4637c408d83f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 34263 } }
I20260925 02:55:12.590067 26586 ts_tablet_manager.cc:1428] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 02:55:12.590417 26419 heartbeater.cc:503] Master 127.12.222.254:38225 was elected leader, sending a full tablet report...
I20260925 02:55:12.593293 26588 tablet_bootstrap.cc:492] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f: Bootstrap starting.
I20260925 02:55:12.595367 26588 tablet_bootstrap.cc:654] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:12.596194 26588 log.cc:824] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:12.597970 26590 tablet_bootstrap.cc:492] T 65d4fadb2df94ea385f25139915d33f3 P b6df70ed89af480c8e2e4637c408d83f: Bootstrap starting.
I20260925 02:55:12.598158 26588 tablet_bootstrap.cc:492] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f: No bootstrap required, opened a new log
I20260925 02:55:12.598364 26588 ts_tablet_manager.cc:1397] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:55:12.599989 26590 tablet_bootstrap.cc:654] T 65d4fadb2df94ea385f25139915d33f3 P b6df70ed89af480c8e2e4637c408d83f: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:12.600744 26590 log.cc:824] T 65d4fadb2df94ea385f25139915d33f3 P b6df70ed89af480c8e2e4637c408d83f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:12.602626 26588 raft_consensus.cc:348] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f2c0418f0d4179b8160b0eded6988f" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45507 } } peers { permanent_uuid: "e349bfeacbfe4a01aa7f2096da322092" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36629 } } peers { permanent_uuid: "b6df70ed89af480c8e2e4637c408d83f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 34263 } }
I20260925 02:55:12.602862 26588 raft_consensus.cc:374] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:12.602921 26588 raft_consensus.cc:729] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f2c0418f0d4179b8160b0eded6988f, State: Initialized, Role: FOLLOWER
I20260925 02:55:12.603336 26588 consensus_queue.cc:260] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f2c0418f0d4179b8160b0eded6988f" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45507 } } peers { permanent_uuid: "e349bfeacbfe4a01aa7f2096da322092" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36629 } } peers { permanent_uuid: "b6df70ed89af480c8e2e4637c408d83f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 34263 } }
I20260925 02:55:12.604249 26588 ts_tablet_manager.cc:1428] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 02:55:12.604604 26560 heartbeater.cc:503] Master 127.12.222.254:38225 was elected leader, sending a full tablet report...
I20260925 02:55:12.605885 26590 tablet_bootstrap.cc:492] T 65d4fadb2df94ea385f25139915d33f3 P b6df70ed89af480c8e2e4637c408d83f: No bootstrap required, opened a new log
I20260925 02:55:12.606062 26590 ts_tablet_manager.cc:1397] T 65d4fadb2df94ea385f25139915d33f3 P b6df70ed89af480c8e2e4637c408d83f: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260925 02:55:12.609750 26590 raft_consensus.cc:348] T 65d4fadb2df94ea385f25139915d33f3 P b6df70ed89af480c8e2e4637c408d83f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f2c0418f0d4179b8160b0eded6988f" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45507 } } peers { permanent_uuid: "e349bfeacbfe4a01aa7f2096da322092" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36629 } } peers { permanent_uuid: "b6df70ed89af480c8e2e4637c408d83f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 34263 } }
I20260925 02:55:12.609975 26590 raft_consensus.cc:374] T 65d4fadb2df94ea385f25139915d33f3 P b6df70ed89af480c8e2e4637c408d83f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:12.610018 26590 raft_consensus.cc:729] T 65d4fadb2df94ea385f25139915d33f3 P b6df70ed89af480c8e2e4637c408d83f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6df70ed89af480c8e2e4637c408d83f, State: Initialized, Role: FOLLOWER
I20260925 02:55:12.610479 26590 consensus_queue.cc:260] T 65d4fadb2df94ea385f25139915d33f3 P b6df70ed89af480c8e2e4637c408d83f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f2c0418f0d4179b8160b0eded6988f" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45507 } } peers { permanent_uuid: "e349bfeacbfe4a01aa7f2096da322092" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36629 } } peers { permanent_uuid: "b6df70ed89af480c8e2e4637c408d83f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 34263 } }
I20260925 02:55:12.611152 26285 heartbeater.cc:503] Master 127.12.222.254:38225 was elected leader, sending a full tablet report...
I20260925 02:55:12.611423 26590 ts_tablet_manager.cc:1428] T 65d4fadb2df94ea385f25139915d33f3 P b6df70ed89af480c8e2e4637c408d83f: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
Sep 25 02:55:12 dist-test-slave-252r krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304912, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for HTTP/127.12.222.212@KRBTEST.COM
W20260925 02:55:12.744807 26420 tablet.cc:2369] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:55:12.761530 26286 tablet.cc:2369] T 65d4fadb2df94ea385f25139915d33f3 P b6df70ed89af480c8e2e4637c408d83f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:55:12.768791 26561 tablet.cc:2369] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:55:12.888031 26589 raft_consensus.cc:482] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:55:12.888281 26589 raft_consensus.cc:504] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f2c0418f0d4179b8160b0eded6988f" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45507 } } peers { permanent_uuid: "e349bfeacbfe4a01aa7f2096da322092" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36629 } } peers { permanent_uuid: "b6df70ed89af480c8e2e4637c408d83f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 34263 } }
I20260925 02:55:12.889483 26589 leader_election.cc:288] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68f2c0418f0d4179b8160b0eded6988f (127.12.222.195:45507), b6df70ed89af480c8e2e4637c408d83f (127.12.222.193:34263)
I20260925 02:55:12.893524 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.889815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:44753 (local address 127.12.222.195:45507)
0925 02:55:12.889981 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:12.889986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:12.890325 (+ 339us) server_negotiation.cc:407] Connection header received
0925 02:55:12.890423 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:12.890426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:12.890468 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:12.890560 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:12.890871 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.891382 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.892530 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.892950 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.893044 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.893110 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:12.893313 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 02:55:12.893402 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:55:12.893532 26605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.889718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:45507 (local address 127.12.222.194:44753)
0925 02:55:12.890183 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:12.890195 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:12.890336 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:12.890593 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:12.890598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:12.890629 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:12.890751 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.890755 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.891513 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.891517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.892395 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.892403 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.893078 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.893084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.893174 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.893198 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:55:12.893280 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:55:12.893326 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":204,"thread_start_us":102,"threads_started":1}
I20260925 02:55:12.893570 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.889852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:50079 (local address 127.12.222.193:34263)
0925 02:55:12.889981 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:12.889986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:12.890249 (+ 263us) server_negotiation.cc:407] Connection header received
0925 02:55:12.890367 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:12.890372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:12.890428 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:12.890497 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:12.890937 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.891393 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.892632 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.893022 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.893140 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.893189 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:12.893394 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 02:55:12.893470 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 02:55:12.893556 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.889718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:34263 (local address 127.12.222.194:50079)
0925 02:55:12.890130 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:12.890145 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:12.890256 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:12.890539 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:12.890545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:12.890563 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:12.890779 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.890785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.891518 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.891521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.892514 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:12.892525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.893155 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.893158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:12.893252 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.893291 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 02:55:12.893362 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 02:55:12.893416 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":92,"spinlock_wait_cycles":20736,"thread_start_us":91,"threads_started":1}
I20260925 02:55:12.894424 26514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d4fadb2df94ea385f25139915d33f3" candidate_uuid: "e349bfeacbfe4a01aa7f2096da322092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68f2c0418f0d4179b8160b0eded6988f" is_pre_election: true
I20260925 02:55:12.894702 26514 raft_consensus.cc:2454] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e349bfeacbfe4a01aa7f2096da322092 in term 0.
I20260925 02:55:12.895190 26307 leader_election.cc:302] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68f2c0418f0d4179b8160b0eded6988f, e349bfeacbfe4a01aa7f2096da322092; no voters:
I20260925 02:55:12.895234 26235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d4fadb2df94ea385f25139915d33f3" candidate_uuid: "e349bfeacbfe4a01aa7f2096da322092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6df70ed89af480c8e2e4637c408d83f" is_pre_election: true
I20260925 02:55:12.895478 26589 raft_consensus.cc:2781] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:55:12.895551 26589 raft_consensus.cc:482] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:55:12.895490 26235 raft_consensus.cc:2454] T 65d4fadb2df94ea385f25139915d33f3 P b6df70ed89af480c8e2e4637c408d83f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e349bfeacbfe4a01aa7f2096da322092 in term 0.
I20260925 02:55:12.895581 26589 raft_consensus.cc:3037] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:12.896721 26589 raft_consensus.cc:504] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f2c0418f0d4179b8160b0eded6988f" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45507 } } peers { permanent_uuid: "e349bfeacbfe4a01aa7f2096da322092" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36629 } } peers { permanent_uuid: "b6df70ed89af480c8e2e4637c408d83f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 34263 } }
I20260925 02:55:12.897089 26589 leader_election.cc:288] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [CANDIDATE]: Term 1 election: Requested vote from peers 68f2c0418f0d4179b8160b0eded6988f (127.12.222.195:45507), b6df70ed89af480c8e2e4637c408d83f (127.12.222.193:34263)
I20260925 02:55:12.897330 26514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d4fadb2df94ea385f25139915d33f3" candidate_uuid: "e349bfeacbfe4a01aa7f2096da322092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68f2c0418f0d4179b8160b0eded6988f"
I20260925 02:55:12.897382 26235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d4fadb2df94ea385f25139915d33f3" candidate_uuid: "e349bfeacbfe4a01aa7f2096da322092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6df70ed89af480c8e2e4637c408d83f"
I20260925 02:55:12.897447 26514 raft_consensus.cc:3037] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:12.897478 26235 raft_consensus.cc:3037] T 65d4fadb2df94ea385f25139915d33f3 P b6df70ed89af480c8e2e4637c408d83f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:12.898553 26514 raft_consensus.cc:2454] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e349bfeacbfe4a01aa7f2096da322092 in term 1.
I20260925 02:55:12.898553 26235 raft_consensus.cc:2454] T 65d4fadb2df94ea385f25139915d33f3 P b6df70ed89af480c8e2e4637c408d83f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e349bfeacbfe4a01aa7f2096da322092 in term 1.
I20260925 02:55:12.898787 26307 leader_election.cc:302] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68f2c0418f0d4179b8160b0eded6988f, e349bfeacbfe4a01aa7f2096da322092; no voters:
I20260925 02:55:12.899026 26589 raft_consensus.cc:2781] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:12.899180 26589 raft_consensus.cc:686] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [term 1 LEADER]: Becoming Leader. State: Replica: e349bfeacbfe4a01aa7f2096da322092, State: Running, Role: LEADER
I20260925 02:55:12.899533 26589 consensus_queue.cc:237] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f2c0418f0d4179b8160b0eded6988f" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45507 } } peers { permanent_uuid: "e349bfeacbfe4a01aa7f2096da322092" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36629 } } peers { permanent_uuid: "b6df70ed89af480c8e2e4637c408d83f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 34263 } }
I20260925 02:55:12.902621 26087 catalog_manager.cc:5495] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 reported cstate change: term changed from 0 to 1, leader changed from <none> to e349bfeacbfe4a01aa7f2096da322092 (127.12.222.194). New cstate: current_term: 1 leader_uuid: "e349bfeacbfe4a01aa7f2096da322092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f2c0418f0d4179b8160b0eded6988f" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e349bfeacbfe4a01aa7f2096da322092" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6df70ed89af480c8e2e4637c408d83f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 34263 } health_report { overall_health: UNKNOWN } } }
I20260925 02:55:12.919526 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.916188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.12.222.194:36629)
0925 02:55:12.916319 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:12.916325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:12.916362 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:55:12.916577 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:12.916580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:12.916658 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:12.916781 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:55:12.917004 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.917476 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.918228 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:12.918406 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:12.918498 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:12.918727 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:55:12.919223 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:55:12.919313 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:12.919377 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 02:55:12.919419 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:55:12.924962 26514 raft_consensus.cc:1261] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f [term 1 FOLLOWER]: Refusing update from remote peer e349bfeacbfe4a01aa7f2096da322092: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:12.925009 26235 raft_consensus.cc:1261] T 65d4fadb2df94ea385f25139915d33f3 P b6df70ed89af480c8e2e4637c408d83f [term 1 FOLLOWER]: Refusing update from remote peer e349bfeacbfe4a01aa7f2096da322092: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:12.925711 26589 consensus_queue.cc:1035] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6df70ed89af480c8e2e4637c408d83f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 34263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:12.925884 26606 consensus_queue.cc:1035] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68f2c0418f0d4179b8160b0eded6988f" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:12.932860 26609 mvcc.cc:205] Tried to move back new op lower bound from 7333088923333570560 to 7333088923238608896. Current Snapshot: MvccSnapshot[applied={T|T < 7333088923333570560}]
I20260925 02:55:12.933367 26610 mvcc.cc:205] Tried to move back new op lower bound from 7333088923333570560 to 7333088923238608896. Current Snapshot: MvccSnapshot[applied={T|T < 7333088923333570560}]
I20260925 02:55:12.936023 26611 mvcc.cc:205] Tried to move back new op lower bound from 7333088923333570560 to 7333088923238608896. Current Snapshot: MvccSnapshot[applied={T|T < 7333088923333570560 or (T in {7333088923333570560})}]
I20260925 02:55:12.942436 26087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:12.942556 26087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:12.944695 26087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eec9fdd8beac4f659ede216819aff54a: Sending DeleteTablet for 3 replicas of tablet 65d4fadb2df94ea385f25139915d33f3
I20260925 02:55:12.945453 26494 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d4fadb2df94ea385f25139915d33f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:12 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:52038
I20260925 02:55:12.945497 26215 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d4fadb2df94ea385f25139915d33f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:12 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:52626
I20260925 02:55:12.945910 26619 tablet_replica.cc:330] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f: stopping tablet replica
I20260925 02:55:12.946091 26340 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d4fadb2df94ea385f25139915d33f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:12 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:52188
I20260925 02:55:12.946220 26620 tablet_replica.cc:330] T 65d4fadb2df94ea385f25139915d33f3 P b6df70ed89af480c8e2e4637c408d83f: stopping tablet replica
I20260925 02:55:12.946352 26619 raft_consensus.cc:2229] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:12.946507 26620 raft_consensus.cc:2229] T 65d4fadb2df94ea385f25139915d33f3 P b6df70ed89af480c8e2e4637c408d83f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:12.946707 26619 raft_consensus.cc:2258] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:12.946719 26620 raft_consensus.cc:2258] T 65d4fadb2df94ea385f25139915d33f3 P b6df70ed89af480c8e2e4637c408d83f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:12.946841 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 26125
I20260925 02:55:12.948127 26619 ts_tablet_manager.cc:1905] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:12.950173 26619 ts_tablet_manager.cc:1918] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:12.950277 26619 log.cc:1196] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/65d4fadb2df94ea385f25139915d33f3
I20260925 02:55:12.950650 26619 ts_tablet_manager.cc:1939] T 65d4fadb2df94ea385f25139915d33f3 P 68f2c0418f0d4179b8160b0eded6988f: Deleting consensus metadata
I20260925 02:55:12.951691 26072 catalog_manager.cc:4841] TS 68f2c0418f0d4179b8160b0eded6988f (127.12.222.195:45507): tablet 65d4fadb2df94ea385f25139915d33f3 (table test-table [id=f86836ea86b1462bbf056bdefe7b9207]) successfully deleted
I20260925 02:55:12.952421 26621 tablet_replica.cc:330] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092: stopping tablet replica
I20260925 02:55:12.952683 26621 raft_consensus.cc:2229] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:55:12.952975 26621 raft_consensus.cc:2258] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:12.954078 26621 ts_tablet_manager.cc:1905] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:12.956427 26621 ts_tablet_manager.cc:1918] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:12.956533 26621 log.cc:1196] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/65d4fadb2df94ea385f25139915d33f3
W20260925 02:55:12.956548 26071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.222.193:34263 (error 108)
I20260925 02:55:12.956856 26621 ts_tablet_manager.cc:1939] T 65d4fadb2df94ea385f25139915d33f3 P e349bfeacbfe4a01aa7f2096da322092: Deleting consensus metadata
I20260925 02:55:12.957262 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 26289
I20260925 02:55:12.957573 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.957284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:34263 (local address 127.0.0.1:52632)
0925 02:55:12.957434 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:12.957498 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.193:34263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260925 02:55:12.958041 26071 catalog_manager.cc:4584] TS b6df70ed89af480c8e2e4637c408d83f (127.12.222.193:34263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 65d4fadb2df94ea385f25139915d33f3: Network error: Client connection negotiation failed: client connection to 127.12.222.193:34263: connect: Connection refused (error 111)
I20260925 02:55:12.962838 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 26423
I20260925 02:55:12.963690 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:12.963030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:36629 (local address 127.0.0.1:52200)
0925 02:55:12.963554 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:12.963611 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.194:36629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440}
I20260925 02:55:12.967975 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 26054
I20260925 02:55:12.974542 13179 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 02:55:13.319599 13179 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 02:55:13.319703 13179 mini_ranger.cc:66] Stopping Ranger...
I20260925 02:55:13.665303 13179 mini_ranger.cc:68] Stopped Ranger
2026-09-25 02:55:13.665 UTC [25120] LOG: received smart shutdown request
2026-09-25 02:55:13.669 UTC [25120] LOG: background worker "logical replication launcher" (PID 25129) exited with exit code 1
2026-09-25 02:55:13.670 UTC [25124] LOG: shutting down
2026-09-25 02:55:13.819 UTC [25120] LOG: database system is shut down
2026-09-25T02:55:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (44750 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:13 dist-test-slave-252r krb5kdc[26631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:13 dist-test-slave-252r krb5kdc[26631](info): set up 2 sockets
Sep 25 02:55:13 dist-test-slave-252r krb5kdc[26631](info): commencing operation
krb5kdc: starting...
W20260925 02:55:14.908162 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:55:14 dist-test-slave-252r krb5kdc[26631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:55:15.018951 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:39829
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:39829,127.12.222.253:44929,127.12.222.252:44281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:15.120702 26644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:15.120945 26644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:15.120982 26644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:15.124655 26644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:15.124694 26644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:15.124711 26644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:15.124725 26644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:15.124773 26644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:15.129554 26644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:39829,127.12.222.253:44929,127.12.222.252:44281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:39829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:15.130391 26644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:15.131212 26644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:55:15.138756 26644 server_base.cc:1034] running on GCE node
W20260925 02:55:15.138736 26652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:15.138720 26649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:15.138918 26650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:15.139415 26644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:15.140363 26644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:15.141539 26644 hybrid_clock.cc:648] HybridClock initialized: now 1790304915141521 us; error 27 us; skew 500 ppm
Sep 25 02:55:15 dist-test-slave-252r krb5kdc[26631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304915, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:15.144014 26644 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:55:15.144767 26644 webserver.cc:460] Webserver started at http://127.12.222.254:42373/ using document root <none> and password file <none>
I20260925 02:55:15.145200 26644 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:15.145251 26644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:15.145397 26644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:15.147150 26644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "ebb242eac42047098e37635b8651e2a5"
format_stamp: "Formatted at 2026-09-25 02:55:15 on dist-test-slave-252r"
server_key: "f4fc68859b75fd872d2a3eba32cdfb3f"
server_key_iv: "eaef3c2d0d568b25f9c50db4bc9c8612"
server_key_version: "encryptionkey@0"
I20260925 02:55:15.147609 26644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "ebb242eac42047098e37635b8651e2a5"
format_stamp: "Formatted at 2026-09-25 02:55:15 on dist-test-slave-252r"
server_key: "f4fc68859b75fd872d2a3eba32cdfb3f"
server_key_iv: "eaef3c2d0d568b25f9c50db4bc9c8612"
server_key_version: "encryptionkey@0"
I20260925 02:55:15.150736 26644 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 02:55:15.152794 26659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:15.153861 26644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:55:15.153986 26644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "ebb242eac42047098e37635b8651e2a5"
format_stamp: "Formatted at 2026-09-25 02:55:15 on dist-test-slave-252r"
server_key: "f4fc68859b75fd872d2a3eba32cdfb3f"
server_key_iv: "eaef3c2d0d568b25f9c50db4bc9c8612"
server_key_version: "encryptionkey@0"
I20260925 02:55:15.154078 26644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:15.165534 26644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:15.167790 26644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:15.167960 26644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:15.173987 26644 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:39829
I20260925 02:55:15.174273 26711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:39829 every 8 connection(s)
I20260925 02:55:15.174820 26644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:15.177294 26712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:15.180364 26712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:55:15.184109 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 26644
I20260925 02:55:15.184233 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:55:15.184573 13179 external_mini_cluster.cc:1428] Setting key ded642afb15fd7ad0700149018e7d115
Sep 25 02:55:15 dist-test-slave-252r krb5kdc[26631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304915, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.222.253@KRBTEST.COM; defaulting to no policy
I20260925 02:55:15.194211 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.182732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51100 (local address 127.12.222.254:39829)
0925 02:55:15.182986 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:15.182994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:15.183025 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:55:15.183437 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:15.183441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:15.183591 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:15.183721 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:15.184526 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.185769 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.186713 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.187350 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.187540 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.190003 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:15.190008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:15.190015 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:15.190035 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:15.191507 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.192020 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.192024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.192025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.192114 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.192455 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.192458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.192459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.192740 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:15.192834 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:15.193292 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 02:55:15.193453 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"spinlock_wait_cycles":11008,"thread_start_us":95,"threads_started":1}
I20260925 02:55:15.194211 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.182125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:39829 (local address 127.0.0.1:51100)
0925 02:55:15.182509 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.182546 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:15.183201 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:15.183753 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:15.183756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:15.184050 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:15.184376 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.184391 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.185884 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.185887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.186586 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.186595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.187506 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.187508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.187583 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.188142 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:15.188157 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:15.189841 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:15.191617 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:15.191629 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:15.191640 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:15.191922 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:15.192205 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:15.192208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:15.192209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:15.192368 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:15.192865 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:15.192874 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:15.193051 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 02:55:15.193230 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 02:55:15.193532 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":77,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.253@KRBTEST.COM" created.
I20260925 02:55:15.196395 26712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:55:15.197079 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.196857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:44929 (local address 127.0.0.1:40920)
0925 02:55:15.196963 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.197021 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 02:55:15.197297 26662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44929: connect: Connection refused (error 111)
I20260925 02:55:15.198076 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.197899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:44929 (local address 127.0.0.1:40932)
0925 02:55:15.197994 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.198033 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 02:55:15.198226 26712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.253:44929: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44929: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.253.keytab.
Entry for principal kudu/127.12.222.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.253.keytab.
I20260925 02:55:15.209378 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.253:44929
--webserver_interface=127.12.222.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.253.keytab
--principal=kudu/127.12.222.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:39829,127.12.222.253:44929,127.12.222.252:44281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:55:15.247490 26712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } attempt: 1
I20260925 02:55:15.248214 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.247973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:44929 (local address 127.0.0.1:40944)
0925 02:55:15.248098 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.248153 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 02:55:15.248915 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.248716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:44929 (local address 127.0.0.1:40952)
0925 02:55:15.248811 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.248854 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 02:55:15.249032 26712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.253:44929: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44929: connect: Connection refused (error 111)
W20260925 02:55:15.308589 26719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:15.308777 26719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:15.308820 26719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:15.312301 26719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:15.312357 26719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:15.312383 26719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:15.312398 26719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:15.312412 26719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:15.317060 26719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:39829,127.12.222.253:44929,127.12.222.252:44281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.253.keytab
--principal=kudu/127.12.222.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.253:44929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.222.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:15.317229 26712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } attempt: 2
I20260925 02:55:15.317952 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.317661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:44929 (local address 127.0.0.1:40960)
0925 02:55:15.317816 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.317878 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 02:55:15.318018 26719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:15.318603 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.318400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:44929 (local address 127.0.0.1:40968)
0925 02:55:15.318505 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.318551 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 02:55:15.318723 26712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.253:44929: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44929: connect: Connection refused (error 111)
I20260925 02:55:15.319052 26719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:15.326682 26727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:15.326704 26724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:15.326686 26725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:15.327075 26719 server_base.cc:1034] running on GCE node
I20260925 02:55:15.327574 26719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:15.328522 26719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:15.329667 26719 hybrid_clock.cc:648] HybridClock initialized: now 1790304915329656 us; error 20 us; skew 500 ppm
Sep 25 02:55:15 dist-test-slave-252r krb5kdc[26631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304915, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:15.332207 26719 init.cc:377] Logged in from keytab as kudu/127.12.222.253@KRBTEST.COM (short username kudu)
I20260925 02:55:15.333041 26719 webserver.cc:460] Webserver started at http://127.12.222.253:41191/ using document root <none> and password file <none>
I20260925 02:55:15.333496 26719 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:15.333555 26719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:15.333774 26719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:15.335546 26719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-1/data/instance:
uuid: "ee22994678824f528425e2bfa7fc5e50"
format_stamp: "Formatted at 2026-09-25 02:55:15 on dist-test-slave-252r"
server_key: "610989b6b9c0bf3fe624b6457e316654"
server_key_iv: "98081d5802ae1d6ac957148bc2aeb32b"
server_key_version: "encryptionkey@0"
I20260925 02:55:15.336093 26719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-1/wal/instance:
uuid: "ee22994678824f528425e2bfa7fc5e50"
format_stamp: "Formatted at 2026-09-25 02:55:15 on dist-test-slave-252r"
server_key: "610989b6b9c0bf3fe624b6457e316654"
server_key_iv: "98081d5802ae1d6ac957148bc2aeb32b"
server_key_version: "encryptionkey@0"
I20260925 02:55:15.339305 26719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260925 02:55:15.341483 26734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:15.342480 26719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:55:15.342590 26719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-1/wal
uuid: "ee22994678824f528425e2bfa7fc5e50"
format_stamp: "Formatted at 2026-09-25 02:55:15 on dist-test-slave-252r"
server_key: "610989b6b9c0bf3fe624b6457e316654"
server_key_iv: "98081d5802ae1d6ac957148bc2aeb32b"
server_key_version: "encryptionkey@0"
I20260925 02:55:15.342676 26719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:15.358768 26719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:15.361573 26719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:15.361755 26719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:15.367986 26719 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.253:44929
I20260925 02:55:15.368295 26786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.253:44929 every 8 connection(s)
I20260925 02:55:15.368847 26719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-1/data/info.pb
I20260925 02:55:15.371052 26787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:15.374212 26787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:55:15.374701 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 26719
I20260925 02:55:15.374792 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-1/wal/instance
I20260925 02:55:15.375059 13179 external_mini_cluster.cc:1428] Setting key 4b23a39c93ea9515cc0e9c6f541b4c7e
WARNING: no policy specified for kudu/127.12.222.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.252@KRBTEST.COM" created.
Sep 25 02:55:15 dist-test-slave-252r krb5kdc[26631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304915, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.253@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:15.389637 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.376343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:39829 (local address 127.0.0.1:51102)
0925 02:55:15.376865 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.376928 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:15.377959 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:15.378470 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:15.378475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:15.378741 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:15.379220 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.379232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.379950 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.379958 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.380731 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.380740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.381237 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.381240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.381407 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.381947 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:15.381962 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:15.385597 (+ 3635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:15.387474 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:15.387479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:15.387491 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:15.387714 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:15.387986 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:15.387989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:15.387991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:15.388113 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:15.388538 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:15.388545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:15.388750 (+ 205us) client_negotiation.cc:769] Sending connection context
0925 02:55:15.388866 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 02:55:15.389055 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":124,"threads_started":1}
I20260925 02:55:15.390234 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.375905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.12.222.254:39829)
0925 02:55:15.376033 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:15.376037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:15.377038 (+ 1001us) server_negotiation.cc:407] Connection header received
0925 02:55:15.378167 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:15.378171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:15.378224 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:15.378319 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:15.379351 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.379823 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.380873 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.381099 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.381202 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.385757 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:15.385766 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:15.385768 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:15.385789 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:15.387367 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.387817 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.387820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.387822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.387900 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.388211 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.388214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.388216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.388395 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:15.389975 (+ 1580us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:15.390058 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 02:55:15.390141 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:55:15.391309 26787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:55:15 dist-test-slave-252r krb5kdc[26631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304915, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.253@KRBTEST.COM for kudu/127.12.222.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.252.keytab.
Entry for principal kudu/127.12.222.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.252.keytab.
I20260925 02:55:15.399386 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.252:44281
--webserver_interface=127.12.222.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.252.keytab
--principal=kudu/127.12.222.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:39829,127.12.222.253:44929,127.12.222.252:44281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:55:15.399979 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.391704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:44929 (local address 127.0.0.1:40974)
0925 02:55:15.391798 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.391811 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:15.391877 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:15.392377 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:15.392380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:15.392525 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:15.392604 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.392609 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.393631 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.393637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.394220 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.394233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.394937 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.394943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.395038 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.395402 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:15.395414 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:15.396572 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:15.398326 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:15.398329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:15.398348 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:15.398582 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:15.398926 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:15.398929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:15.398932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:15.399026 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:15.399403 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:15.399408 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:15.399507 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 02:55:15.399614 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:55:15.399746 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 02:55:15.400544 26792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.391835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40974 (local address 127.12.222.253:44929)
0925 02:55:15.392066 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:15.392077 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:15.392093 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:55:15.392120 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:15.392123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:15.392283 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:15.392395 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:15.392722 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.393526 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.394328 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.394781 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.394901 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.396764 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:15.396771 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:15.396779 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:15.396799 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:15.398234 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.398702 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.398710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.398714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.398806 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.399116 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.399118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.399119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.399299 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:15.399387 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:15.400194 (+ 807us) server_negotiation.cc:299] Negotiation successful
0925 02:55:15.400355 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":75,"threads_started":1}
I20260925 02:55:15.401952 26787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:55:15.402556 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.402330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44281 (local address 127.0.0.1:33278)
0925 02:55:15.402446 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.402511 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 02:55:15.402750 26736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
I20260925 02:55:15.403502 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.403316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44281 (local address 127.0.0.1:33288)
0925 02:55:15.403427 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.403463 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 02:55:15.403728 26787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.252:44281: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
I20260925 02:55:15.409868 26712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } attempt: 3
Sep 25 02:55:15 dist-test-slave-252r krb5kdc[26631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304915, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for kudu/127.12.222.253@KRBTEST.COM
I20260925 02:55:15.417272 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.410191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:44929 (local address 127.0.0.1:40984)
0925 02:55:15.410317 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.410329 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:15.410398 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:15.410590 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:15.410592 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:15.410732 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:15.410809 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.410814 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.411410 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.411413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.412342 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.412352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.412817 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.412821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.412937 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.413350 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:15.413360 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:15.414385 (+ 1025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:15.415856 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:15.415859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:15.415861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:15.416046 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:15.416367 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:15.416370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:15.416371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:15.416455 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:15.416797 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:15.416800 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:15.416893 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 02:55:15.417001 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 02:55:15.417127 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 02:55:15.417270 26792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.410268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40984 (local address 127.12.222.253:44929)
0925 02:55:15.410419 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:15.410425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:15.410437 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:55:15.410468 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:15.410470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:15.410504 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:15.410573 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:15.410908 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.411307 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.412452 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.412698 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.412805 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.414505 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:15.414510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:15.414517 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:15.414536 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:15.415745 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.416159 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.416161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.416162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.416265 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.416545 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.416549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.416551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.416702 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:15.416801 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:15.417052 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 02:55:15.417148 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 02:55:15.418565 26712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:55:15.419224 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.418998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44281 (local address 127.0.0.1:33290)
0925 02:55:15.419117 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.419167 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 02:55:15.419853 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.419600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44281 (local address 127.0.0.1:33296)
0925 02:55:15.419722 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.419769 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 02:55:15.419994 26712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.252:44281: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
I20260925 02:55:15.451151 26712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } attempt: 1
I20260925 02:55:15.451691 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.451466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44281 (local address 127.0.0.1:33312)
0925 02:55:15.451580 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.451631 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 02:55:15.452358 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.452128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44281 (local address 127.0.0.1:33316)
0925 02:55:15.452247 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.452293 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 02:55:15.452495 26712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.252:44281: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
I20260925 02:55:15.452978 26787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } attempt: 1
I20260925 02:55:15.453498 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.453286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44281 (local address 127.0.0.1:33328)
0925 02:55:15.453393 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.453447 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 02:55:15.454121 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.453947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44281 (local address 127.0.0.1:33340)
0925 02:55:15.454039 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.454078 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 02:55:15.454254 26787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.252:44281: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
W20260925 02:55:15.503670 26793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:15.503855 26793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:15.503959 26793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:15.507735 26793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:15.507777 26793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:15.507794 26793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:15.507853 26793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:15.507891 26793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:15.512346 26793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:39829,127.12.222.253:44929,127.12.222.252:44281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.252.keytab
--principal=kudu/127.12.222.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.252:44281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.222.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:15.513105 26793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:15.513900 26793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:15.521041 26802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:15.521059 26799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:15.521041 26800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:15.521247 26793 server_base.cc:1034] running on GCE node
I20260925 02:55:15.521802 26793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:15.522413 26787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } attempt: 2
I20260925 02:55:15.522954 26793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:15.523065 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.522775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44281 (local address 127.0.0.1:33356)
0925 02:55:15.522933 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.522988 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 02:55:15.524005 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.523822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44281 (local address 127.0.0.1:33370)
0925 02:55:15.523917 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.523959 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 02:55:15.524139 26787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.252:44281: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
I20260925 02:55:15.524163 26793 hybrid_clock.cc:648] HybridClock initialized: now 1790304915524136 us; error 38 us; skew 500 ppm
Sep 25 02:55:15 dist-test-slave-252r krb5kdc[26631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304915, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:15.526664 26793 init.cc:377] Logged in from keytab as kudu/127.12.222.252@KRBTEST.COM (short username kudu)
I20260925 02:55:15.527393 26793 webserver.cc:460] Webserver started at http://127.12.222.252:35533/ using document root <none> and password file <none>
I20260925 02:55:15.527662 26712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } attempt: 2
I20260925 02:55:15.527952 26793 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:15.528015 26793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:15.528210 26793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:15.528316 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.528021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44281 (local address 127.0.0.1:33374)
0925 02:55:15.528168 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.528207 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 02:55:15.529007 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.528799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44281 (local address 127.0.0.1:33388)
0925 02:55:15.528901 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.528944 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 02:55:15.529155 26712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.252:44281: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44281: connect: Connection refused (error 111)
I20260925 02:55:15.530274 26793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-2/data/instance:
uuid: "4dc59a88ad25484f824fae985faf4339"
format_stamp: "Formatted at 2026-09-25 02:55:15 on dist-test-slave-252r"
server_key: "b28c9555d3f95b415c44154db31f0308"
server_key_iv: "37bf3febb9e6c642bbb4fef919af1a63"
server_key_version: "encryptionkey@0"
I20260925 02:55:15.530738 26793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-2/wal/instance:
uuid: "4dc59a88ad25484f824fae985faf4339"
format_stamp: "Formatted at 2026-09-25 02:55:15 on dist-test-slave-252r"
server_key: "b28c9555d3f95b415c44154db31f0308"
server_key_iv: "37bf3febb9e6c642bbb4fef919af1a63"
server_key_version: "encryptionkey@0"
I20260925 02:55:15.534010 26793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 02:55:15.536106 26809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:15.537202 26793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:55:15.537348 26793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-2/wal
uuid: "4dc59a88ad25484f824fae985faf4339"
format_stamp: "Formatted at 2026-09-25 02:55:15 on dist-test-slave-252r"
server_key: "b28c9555d3f95b415c44154db31f0308"
server_key_iv: "37bf3febb9e6c642bbb4fef919af1a63"
server_key_version: "encryptionkey@0"
I20260925 02:55:15.537446 26793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:15.569553 26793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:15.571945 26793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:15.572124 26793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:15.578260 26793 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.252:44281
I20260925 02:55:15.578498 26861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.252:44281 every 8 connection(s)
I20260925 02:55:15.579106 26793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-2/data/info.pb
I20260925 02:55:15.581501 26862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:15.584741 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 26793
I20260925 02:55:15.584856 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304762911399-13179-0/minicluster-data/master-2/wal/instance
I20260925 02:55:15.584667 26862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:55:15.585152 13179 external_mini_cluster.cc:1428] Setting key 98a6bf7ff9d3716b766e3f6799352922
Sep 25 02:55:15 dist-test-slave-252r krb5kdc[26631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304915, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.252@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
Sep 25 02:55:15 dist-test-slave-252r krb5kdc[26631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.252@KRBTEST.COM
I20260925 02:55:15.596431 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.586892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.12.222.254:39829)
0925 02:55:15.587003 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:15.587007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:15.587221 (+ 214us) server_negotiation.cc:407] Connection header received
0925 02:55:15.587618 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:15.587621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:15.587662 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:15.587725 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:15.588612 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.589124 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.590131 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.590398 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.590469 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.592812 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:15.592815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:15.592819 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:15.592846 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:15.594387 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.594971 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.594974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.594976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.595057 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.595383 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.595388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.595390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.595570 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:15.595679 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:15.596147 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 02:55:15.596281 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:55:15.597087 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.586283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.12.222.252:44281)
0925 02:55:15.586621 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:15.586633 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:15.586661 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:55:15.587428 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:15.587448 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:15.587652 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:15.587822 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:15.588510 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.589518 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.590303 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.590716 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.590823 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.593210 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:15.593216 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:15.593223 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:15.593241 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:15.594793 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.595216 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.595220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.595221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.595292 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.595582 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.595586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.595587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.595924 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:15.596112 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:15.596334 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 02:55:15.596675 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":223,"thread_start_us":93,"threads_started":1}
I20260925 02:55:15.597086 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.586774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:39829 (local address 127.0.0.1:51104)
0925 02:55:15.587094 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.587139 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:15.587428 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:15.587791 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:15.587796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:15.588189 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:15.588476 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.588494 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.589218 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.589221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.590024 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.590032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.590517 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.590521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.590655 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.591183 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:15.591196 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:15.592684 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:15.594506 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:15.594514 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:15.594523 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:15.594872 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:15.595156 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:15.595159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:15.595161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:15.595290 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:15.595679 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:15.595686 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:15.595937 (+ 251us) client_negotiation.cc:769] Sending connection context
0925 02:55:15.596088 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 02:55:15.596375 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":42,"spinlock_wait_cycles":3456,"thread_start_us":75,"threads_started":1}
I20260925 02:55:15.598323 26862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:55:15 dist-test-slave-252r krb5kdc[26631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304915, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.252@KRBTEST.COM for kudu/127.12.222.253@KRBTEST.COM
Sep 25 02:55:15 dist-test-slave-252r krb5kdc[26631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:15.606734 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.598751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:44929 (local address 127.0.0.1:40990)
0925 02:55:15.598898 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.598908 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:15.598963 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:15.599422 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:15.599425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:15.599573 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:15.599664 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.599669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.600518 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.600521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.601050 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.601058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.601539 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.601542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.601637 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.602052 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:15.602066 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:15.603307 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:15.605186 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:15.605189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:15.605191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:15.605435 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:15.605714 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:15.605716 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:15.605717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:15.605823 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:15.606270 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:15.606273 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:15.606350 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 02:55:15.606463 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 02:55:15.606584 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 02:55:15.606805 26792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.599073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40990 (local address 127.12.222.253:44929)
0925 02:55:15.599211 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:15.599215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:15.599228 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:55:15.599268 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:15.599270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:15.599316 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:15.599389 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:15.599776 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.600429 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.601153 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.601446 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.601514 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.603444 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:15.603447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:15.603449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:15.603467 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:15.605088 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.605539 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.605542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.605544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.605622 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.605941 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.605945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.605946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.606170 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:15.606248 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:15.606527 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 02:55:15.606638 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:55:15.606976 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.599208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51106 (local address 127.12.222.254:39829)
0925 02:55:15.599294 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:15.599297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:15.599311 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:55:15.599402 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:15.599404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:15.599438 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:15.599518 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:15.600116 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.600622 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.601352 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.601573 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.601661 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.603962 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:15.603965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:15.603968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:15.603990 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:15.605307 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.605778 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.605783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.605784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.605854 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.606131 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.606133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.606134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.606325 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:15.606526 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:15.606708 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 02:55:15.606810 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:55:15.607427 26862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:55:15 dist-test-slave-252r krb5kdc[26631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304915, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.252@KRBTEST.COM for kudu/127.12.222.252@KRBTEST.COM
Sep 25 02:55:15 dist-test-slave-252r krb5kdc[26631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.253@KRBTEST.COM
I20260925 02:55:15.614955 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.607837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44281 (local address 127.0.0.1:33412)
0925 02:55:15.607924 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.607937 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:15.608003 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:15.608463 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:15.608466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:15.608608 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:15.608655 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.608659 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.609264 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.609268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.609705 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.609710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.610113 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.610116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.610178 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.610551 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:15.610562 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:15.611803 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:15.613582 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:15.613584 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:15.613585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:15.613748 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:15.614045 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:15.614047 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:15.614048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:15.614115 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:15.614505 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:15.614510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:15.614596 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 02:55:15.614703 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:55:15.614787 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 02:55:15.615171 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.607974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.12.222.252:44281)
0925 02:55:15.608084 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:15.608088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:15.608101 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:55:15.608133 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:15.608135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:15.608173 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:15.608456 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:15.608751 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.609169 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.609801 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.610009 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.610103 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.611919 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:15.611922 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:15.611924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:15.611943 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:15.613475 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.613858 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.613864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.613867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.613952 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.614205 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.614210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.614212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.614402 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:15.614499 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:15.614781 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 02:55:15.614894 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 02:55:15.615264 26787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } attempt: 3
I20260925 02:55:15.616173 26792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.608174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40992 (local address 127.12.222.253:44929)
0925 02:55:15.608331 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:15.608334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:15.608362 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:55:15.608483 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:15.608486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:15.608520 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:15.608601 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:15.609100 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.609505 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.610137 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.610327 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.610419 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.612638 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:15.612641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:15.612643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:15.612664 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:15.613995 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.614452 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.614455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.614473 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.614555 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.614868 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.614871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.614873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.615083 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:15.615859 (+ 776us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:15.615938 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 02:55:15.616043 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 02:55:15.618597 26862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339: Bootstrap starting.
I20260925 02:55:15.619817 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 25 02:55:15 dist-test-slave-252r krb5kdc[26631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304915, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.253@KRBTEST.COM for kudu/127.12.222.252@KRBTEST.COM
I20260925 02:55:15.621381 26862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:15.622169 26862 log.cc:824] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:15.624004 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.615881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33414 (local address 127.12.222.252:44281)
0925 02:55:15.615970 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:15.615973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:15.616004 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:55:15.616034 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:15.616036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:15.616068 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:15.616140 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:15.616556 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.617191 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.618202 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.618518 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.618627 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.620517 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:15.620520 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:15.620523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:15.620544 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:15.622181 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.622730 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.622733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.622735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.622814 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.623141 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.623145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.623150 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.623386 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:15.623492 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:15.623779 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 02:55:15.623876 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:55:15.624020 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.615652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44281 (local address 127.0.0.1:33414)
0925 02:55:15.615743 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.615758 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:15.615820 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:15.616176 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:15.616181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:15.616371 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:15.616459 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.616466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.617309 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.617313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.618081 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.618091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.618639 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.618643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.618725 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.619121 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:15.619135 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:15.620398 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:15.622294 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:15.622300 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:15.622303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:15.622599 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:15.622929 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:15.622934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:15.622937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:15.623026 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:15.623494 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:15.623498 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:15.623592 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 02:55:15.623705 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 02:55:15.623825 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 02:55:15.627027 26787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50: Bootstrap starting.
I20260925 02:55:15.628307 26712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } attempt: 3
I20260925 02:55:15.629110 26862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339: No bootstrap required, opened a new log
I20260925 02:55:15.629665 26787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:15.630458 26787 log.cc:824] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:15.631170 26792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.622030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41000 (local address 127.12.222.253:44929)
0925 02:55:15.622124 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:15.622128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:15.622680 (+ 552us) server_negotiation.cc:407] Connection header received
0925 02:55:15.622845 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:15.622851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:15.622898 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:15.622944 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:15.623869 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.624456 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.625608 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.625944 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.626058 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.627729 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:15.627732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:15.627735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:15.627758 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:15.629334 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.629795 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.629798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.629799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.629880 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.630186 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.630189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.630191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.630364 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:15.630481 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:15.630947 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 02:55:15.631038 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:55:15.632334 26787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50: No bootstrap required, opened a new log
I20260925 02:55:15.632721 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.622142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.12.222.254:39829)
0925 02:55:15.622230 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:15.622235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:15.623301 (+ 1066us) server_negotiation.cc:407] Connection header received
0925 02:55:15.623405 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:15.623415 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:15.623452 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:15.623516 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:15.624624 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.625286 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.626352 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.626647 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.626749 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.629040 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:15.629043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:15.629045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:15.629066 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:15.630693 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.631160 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.631163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.631165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.631425 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.631727 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.631730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.631731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.631909 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:15.632029 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:15.632493 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 02:55:15.632598 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:55:15.634285 26862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } }
I20260925 02:55:15.634550 26862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:15.634609 26862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc59a88ad25484f824fae985faf4339, State: Initialized, Role: FOLLOWER
I20260925 02:55:15.635038 26862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } }
Sep 25 02:55:15 dist-test-slave-252r krb5kdc[26631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304915, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for kudu/127.12.222.252@KRBTEST.COM
I20260925 02:55:15.635488 26877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } } }
I20260925 02:55:15.635759 26877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:55:15.636052 26787 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } }
I20260925 02:55:15.636231 26787 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:15.636315 26787 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee22994678824f528425e2bfa7fc5e50, State: Initialized, Role: FOLLOWER
I20260925 02:55:15.636377 26862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:15.636762 26787 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } }
I20260925 02:55:15.637885 26787 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:15.639447 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.628654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44281 (local address 127.0.0.1:33434)
0925 02:55:15.628808 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:15.628820 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:15.628884 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:15.631536 (+ 2652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:15.631539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:15.631700 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:15.631773 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.631780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.632694 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.632697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.633431 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:15.633441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.634017 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.634021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:15.634096 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.634551 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:15.634567 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:15.635851 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:15.637876 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:15.637880 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:15.637885 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:15.638119 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:15.638391 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:15.638393 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:15.638395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:15.638475 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:15.638940 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:15.638943 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:15.639036 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 02:55:15.639132 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 02:55:15.639261 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260925 02:55:15.640606 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.628685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33434 (local address 127.12.222.252:44281)
0925 02:55:15.631320 (+ 2635us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:15.631324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:15.631338 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:55:15.631374 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:15.631377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:15.631414 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:15.631878 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:15.631968 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.632577 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.633572 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.633878 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.634287 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.636005 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:15.636009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:15.636011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:15.636034 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:15.637784 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.638219 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.638222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.638223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.638297 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.638583 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.638586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.638588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.638786 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:15.638879 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:15.640367 (+ 1488us) server_negotiation.cc:299] Negotiation successful
0925 02:55:15.640486 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 02:55:15.640434 26793 master_runner.cc:186] Error getting master registration for 127.12.222.254:39829: OK, instance_id { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" instance_seqno: 1790304915172194 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:55:15.644501 26793 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 02:55:15.645102 26901 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:55:15.645311 26901 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:55:15.642998 26719 master_runner.cc:186] Error getting master registration for 127.12.222.254:39829: OK, instance_id { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" instance_seqno: 1790304915172194 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:55:15.646968 26719 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:55:15.647790 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:15.622859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.12.222.252:44281)
0925 02:55:15.623101 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:15.623107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:15.626177 (+ 3070us) server_negotiation.cc:407] Connection header received
0925 02:55:15.626325 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:15.626329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:15.626375 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:15.626731 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:15.638278 (+ 11547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.638982 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.639983 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:15.640311 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:15.640403 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:15.643653 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:15.643657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:15.643659 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:15.643680 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:15.645543 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.646255 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.646258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.646260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.646364 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:15.646692 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:15.646695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:15.646697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:15.646882 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:15.646996 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:15.647556 (+ 560us) server_negotiation.cc:299] Negotiation successful
0925 02:55:15.647655 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":93,"threads_started":1}
I20260925 02:55:15.647934 26712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5: Bootstrap starting.
I20260925 02:55:15.649895 26712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:15.650506 26712 log.cc:824] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5: Log is configured to *not* fsync() on all Append() calls
W20260925 02:55:15.651330 26889 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:55:15.651392 26889 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:55:15.651757 26878 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } } }
I20260925 02:55:15.651886 26878 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:55:15.653280 26712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5: No bootstrap required, opened a new log
I20260925 02:55:15.655843 26712 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } }
I20260925 02:55:15.656044 26712 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:15.656101 26712 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb242eac42047098e37635b8651e2a5, State: Initialized, Role: FOLLOWER
I20260925 02:55:15.656512 26712 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } }
I20260925 02:55:15.657634 26712 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 02:55:15.662019 26915 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:55:15.662148 26915 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:55:15.665074 26644 master_runner.cc:263] No leader master found from master ebb242eac42047098e37635b8651e2a5
I20260925 02:55:15.665134 26644 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:55:15.670568 26904 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } } }
I20260925 02:55:15.670745 26904 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 02:55:15.674201 26665 tablet.cc:2369] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:55:15.823308 26815 tablet.cc:2369] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:55:15.842345 26878 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:55:15.842557 26878 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } }
I20260925 02:55:15.843382 26878 leader_election.cc:288] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebb242eac42047098e37635b8651e2a5 (127.12.222.254:39829), 4dc59a88ad25484f824fae985faf4339 (127.12.222.252:44281)
I20260925 02:55:15.843932 26836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee22994678824f528425e2bfa7fc5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc59a88ad25484f824fae985faf4339" is_pre_election: true
I20260925 02:55:15.843971 26686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee22994678824f528425e2bfa7fc5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb242eac42047098e37635b8651e2a5" is_pre_election: true
I20260925 02:55:15.844213 26836 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee22994678824f528425e2bfa7fc5e50 in term 0.
I20260925 02:55:15.844295 26686 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee22994678824f528425e2bfa7fc5e50 in term 0.
I20260925 02:55:15.844619 26736 leader_election.cc:302] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc59a88ad25484f824fae985faf4339, ee22994678824f528425e2bfa7fc5e50; no voters:
I20260925 02:55:15.844847 26878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:55:15.844928 26878 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:55:15.844977 26878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:15.846012 26878 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } }
I20260925 02:55:15.846351 26878 leader_election.cc:288] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [CANDIDATE]: Term 1 election: Requested vote from peers ebb242eac42047098e37635b8651e2a5 (127.12.222.254:39829), 4dc59a88ad25484f824fae985faf4339 (127.12.222.252:44281)
I20260925 02:55:15.846627 26686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee22994678824f528425e2bfa7fc5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb242eac42047098e37635b8651e2a5"
I20260925 02:55:15.846675 26836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee22994678824f528425e2bfa7fc5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc59a88ad25484f824fae985faf4339"
I20260925 02:55:15.846792 26686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:15.846805 26836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:15.847683 26836 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee22994678824f528425e2bfa7fc5e50 in term 1.
I20260925 02:55:15.847817 26686 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee22994678824f528425e2bfa7fc5e50 in term 1.
I20260925 02:55:15.847960 26736 leader_election.cc:302] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dc59a88ad25484f824fae985faf4339, ee22994678824f528425e2bfa7fc5e50; no voters:
I20260925 02:55:15.848131 26878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:15.848341 26878 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [term 1 LEADER]: Becoming Leader. State: Replica: ee22994678824f528425e2bfa7fc5e50, State: Running, Role: LEADER
I20260925 02:55:15.848659 26878 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } }
I20260925 02:55:15.850445 26917 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee22994678824f528425e2bfa7fc5e50. Latest consensus state: current_term: 1 leader_uuid: "ee22994678824f528425e2bfa7fc5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } } }
I20260925 02:55:15.850559 26917 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:15.850927 26919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:15.851694 26919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:15.855840 26686 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5 [term 1 FOLLOWER]: Refusing update from remote peer ee22994678824f528425e2bfa7fc5e50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:15.855841 26836 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339 [term 1 FOLLOWER]: Refusing update from remote peer ee22994678824f528425e2bfa7fc5e50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:15.856492 26878 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:15.856680 26917 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:15.860282 26904 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee22994678824f528425e2bfa7fc5e50. Latest consensus state: current_term: 1 leader_uuid: "ee22994678824f528425e2bfa7fc5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } } }
I20260925 02:55:15.860286 26877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee22994678824f528425e2bfa7fc5e50. Latest consensus state: current_term: 1 leader_uuid: "ee22994678824f528425e2bfa7fc5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } } }
I20260925 02:55:15.860409 26904 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:55:15.860419 26877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:55:15.862449 26917 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee22994678824f528425e2bfa7fc5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } } }
I20260925 02:55:15.862491 26878 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee22994678824f528425e2bfa7fc5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } } }
I20260925 02:55:15.862543 26917 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:15.862550 26878 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:15.863272 26904 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee22994678824f528425e2bfa7fc5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } } }
I20260925 02:55:15.863370 26904 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:55:15.863988 26877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee22994678824f528425e2bfa7fc5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb242eac42047098e37635b8651e2a5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 39829 } } peers { permanent_uuid: "ee22994678824f528425e2bfa7fc5e50" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44929 } } peers { permanent_uuid: "4dc59a88ad25484f824fae985faf4339" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44281 } } }
I20260925 02:55:15.864084 26877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:55:15.864548 26919 catalog_manager.cc:1347] Generated new cluster ID: a6d420264e4443ef9d4ed547f0edbbcd
I20260925 02:55:15.864606 26919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:15.873064 26919 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:15.874143 26919 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:15.880649 26919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee22994678824f528425e2bfa7fc5e50: Generated new TSK 0
I20260925 02:55:15.881335 26919 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:55:16.652540 26889 catalog_manager.cc:1259] Loaded cluster ID: a6d420264e4443ef9d4ed547f0edbbcd
I20260925 02:55:16.652609 26889 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339: loading cluster ID for follower catalog manager: success
I20260925 02:55:16.654173 26889 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339: acquiring CA information for follower catalog manager: success
I20260925 02:55:16.655331 26889 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4dc59a88ad25484f824fae985faf4339: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:55:16.663172 26915 catalog_manager.cc:1259] Loaded cluster ID: a6d420264e4443ef9d4ed547f0edbbcd
I20260925 02:55:16.663223 26915 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5: loading cluster ID for follower catalog manager: success
I20260925 02:55:16.664754 26915 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5: acquiring CA information for follower catalog manager: success
I20260925 02:55:16.665683 26915 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ebb242eac42047098e37635b8651e2a5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:55:16.951261 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 26644
I20260925 02:55:16.957491 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 26719
I20260925 02:55:16.962971 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 26793
2026-09-25T02:55:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3027 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:17 dist-test-slave-252r krb5kdc[26935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:17 dist-test-slave-252r krb5kdc[26935](info): set up 2 sockets
Sep 25 02:55:17 dist-test-slave-252r krb5kdc[26935](info): commencing operation
krb5kdc: starting...
W20260925 02:55:17.952391 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:55:18 dist-test-slave-252r krb5kdc[26935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:55:18.062717 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:32899
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:32899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:18.165799 26948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:18.166038 26948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:18.166077 26948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:18.169757 26948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:18.169800 26948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:18.169817 26948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:18.169850 26948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:18.169864 26948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:18.174507 26948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:32899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:32899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:18.175372 26948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:18.176303 26948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:18.184010 26956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:18.184013 26953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:18.184175 26954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:18.184018 26948 server_base.cc:1034] running on GCE node
I20260925 02:55:18.184654 26948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:18.185601 26948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:18.186759 26948 hybrid_clock.cc:648] HybridClock initialized: now 1790304918186743 us; error 28 us; skew 500 ppm
Sep 25 02:55:18 dist-test-slave-252r krb5kdc[26935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304918, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:18.189216 26948 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:55:18.189940 26948 webserver.cc:460] Webserver started at http://127.12.222.254:40923/ using document root <none> and password file <none>
I20260925 02:55:18.190440 26948 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:18.190526 26948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:18.190738 26948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:18.192548 26948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "931aec98c6f94892a0a4ec3de09d116f"
format_stamp: "Formatted at 2026-09-25 02:55:18 on dist-test-slave-252r"
server_key: "b6ca9fcbcd575d73c38ff7660a8ebcb3"
server_key_iv: "1242e704ebea3c8bb5d3c4e9c65d5d86"
server_key_version: "encryptionkey@0"
I20260925 02:55:18.193064 26948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "931aec98c6f94892a0a4ec3de09d116f"
format_stamp: "Formatted at 2026-09-25 02:55:18 on dist-test-slave-252r"
server_key: "b6ca9fcbcd575d73c38ff7660a8ebcb3"
server_key_iv: "1242e704ebea3c8bb5d3c4e9c65d5d86"
server_key_version: "encryptionkey@0"
I20260925 02:55:18.196478 26948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:55:18.198585 26963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:18.199574 26948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:55:18.199738 26948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "931aec98c6f94892a0a4ec3de09d116f"
format_stamp: "Formatted at 2026-09-25 02:55:18 on dist-test-slave-252r"
server_key: "b6ca9fcbcd575d73c38ff7660a8ebcb3"
server_key_iv: "1242e704ebea3c8bb5d3c4e9c65d5d86"
server_key_version: "encryptionkey@0"
I20260925 02:55:18.199839 26948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:18.219871 26948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:18.222402 26948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:18.222586 26948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:18.229086 26948 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:32899
I20260925 02:55:18.229140 27015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:32899 every 8 connection(s)
I20260925 02:55:18.229928 26948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:18.232326 27016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:18.237198 27016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f: Bootstrap starting.
I20260925 02:55:18.237632 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 26948
I20260925 02:55:18.237773 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:55:18.238042 13179 external_mini_cluster.cc:1428] Setting key 9ce0b5e1e77d7759e9a5dd4c20a49699
I20260925 02:55:18.239607 27016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:18.240427 27016 log.cc:824] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:18.242430 27016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f: No bootstrap required, opened a new log
I20260925 02:55:18.246014 27016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931aec98c6f94892a0a4ec3de09d116f" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 32899 } }
I20260925 02:55:18.246261 27016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:18.246328 27016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 931aec98c6f94892a0a4ec3de09d116f, State: Initialized, Role: FOLLOWER
Sep 25 02:55:18 dist-test-slave-252r krb5kdc[26935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:18.246773 27016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931aec98c6f94892a0a4ec3de09d116f" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 32899 } }
I20260925 02:55:18.246958 27016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:18.247028 27016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:18.247138 27016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:18.248139 27016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931aec98c6f94892a0a4ec3de09d116f" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 32899 } }
I20260925 02:55:18.248467 27016 leader_election.cc:302] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 931aec98c6f94892a0a4ec3de09d116f; no voters:
I20260925 02:55:18.248749 27016 leader_election.cc:288] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:18.248811 27021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:18.248996 27021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f [term 1 LEADER]: Becoming Leader. State: Replica: 931aec98c6f94892a0a4ec3de09d116f, State: Running, Role: LEADER
I20260925 02:55:18.249250 27021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931aec98c6f94892a0a4ec3de09d116f" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 32899 } }
I20260925 02:55:18.249842 27016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:18.250243 27021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "931aec98c6f94892a0a4ec3de09d116f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931aec98c6f94892a0a4ec3de09d116f" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 32899 } } }
I20260925 02:55:18.250340 27021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:18.250795 27023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 931aec98c6f94892a0a4ec3de09d116f. Latest consensus state: current_term: 1 leader_uuid: "931aec98c6f94892a0a4ec3de09d116f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931aec98c6f94892a0a4ec3de09d116f" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 32899 } } }
I20260925 02:55:18.250893 27023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:18.252108 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:18.239201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56376 (local address 127.12.222.254:32899)
0925 02:55:18.239590 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:18.239607 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:18.239647 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:55:18.240434 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:18.240457 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:18.240858 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:18.241148 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:18.241798 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:18.243066 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:18.243837 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:18.244508 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:18.244675 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:18.247147 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:18.247154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:18.247170 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:18.247212 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:18.248831 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:18.249314 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:18.249320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:18.249325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:18.249454 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:18.249722 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:18.249725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:18.249726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:18.250138 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:18.250322 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:18.250976 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 02:55:18.251218 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":101,"threads_started":1}
W20260925 02:55:18.254272 27036 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:55:18.254343 27036 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:55:18.254403 27025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:18.254988 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:55:18.255144 27025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:18.260026 27025 catalog_manager.cc:1347] Generated new cluster ID: 6d93e3bd5eb747b9a24cdfa2459ddea1
I20260925 02:55:18.260099 27025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:18.278239 27025 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:18.279351 27025 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:18.291044 27025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 931aec98c6f94892a0a4ec3de09d116f: Generated new TSK 0
I20260925 02:55:18.291790 27025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:55:18.310684 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 26948
2026-09-25T02:55:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1346 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:18 dist-test-slave-252r krb5kdc[27043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:18 dist-test-slave-252r krb5kdc[27043](info): set up 2 sockets
Sep 25 02:55:18 dist-test-slave-252r krb5kdc[27043](info): commencing operation
krb5kdc: starting...
W20260925 02:55:19.275908 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:55:19 dist-test-slave-252r krb5kdc[27043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:55:19.384840 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:34743
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:34743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:19.485193 27056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:19.485430 27056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:19.485496 27056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:19.489362 27056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:19.489434 27056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:19.489468 27056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:19.489518 27056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:19.489550 27056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:19.494145 27056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:34743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:34743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:19.495035 27056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:19.495939 27056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:19.503786 27065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:19.503758 27062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:19.503823 27063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:19.503868 27056 server_base.cc:1034] running on GCE node
I20260925 02:55:19.504602 27056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:19.505568 27056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:19.506732 27056 hybrid_clock.cc:648] HybridClock initialized: now 1790304919506708 us; error 33 us; skew 500 ppm
Sep 25 02:55:19 dist-test-slave-252r krb5kdc[27043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304919, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:19.513643 27056 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:55:19.514433 27056 webserver.cc:460] Webserver started at http://127.12.222.254:37109/ using document root <none> and password file <none>
I20260925 02:55:19.514948 27056 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:19.515008 27056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:19.515235 27056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:19.517194 27056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "00aaad83295f4139b7f27b616d20de64"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-252r"
server_key: "7b8015f0f1e449df50c207b4ed7b0f48"
server_key_iv: "ba18287f4dce18c78d499b5154978176"
server_key_version: "encryptionkey@0"
I20260925 02:55:19.517740 27056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "00aaad83295f4139b7f27b616d20de64"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-252r"
server_key: "7b8015f0f1e449df50c207b4ed7b0f48"
server_key_iv: "ba18287f4dce18c78d499b5154978176"
server_key_version: "encryptionkey@0"
I20260925 02:55:19.521191 27056 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:55:19.523240 27072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:19.524365 27056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:55:19.524499 27056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "00aaad83295f4139b7f27b616d20de64"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-252r"
server_key: "7b8015f0f1e449df50c207b4ed7b0f48"
server_key_iv: "ba18287f4dce18c78d499b5154978176"
server_key_version: "encryptionkey@0"
I20260925 02:55:19.524593 27056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:19.547698 27056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:19.550302 27056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:19.550509 27056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:19.557657 27056 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:34743
I20260925 02:55:19.557915 27124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:34743 every 8 connection(s)
I20260925 02:55:19.558521 27056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:19.560320 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 27056
I20260925 02:55:19.560452 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:55:19.560741 13179 external_mini_cluster.cc:1428] Setting key 51aa3fdadbce63f57ae82d9ec7512562
I20260925 02:55:19.561308 27125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:19.567472 27125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64: Bootstrap starting.
Sep 25 02:55:19 dist-test-slave-252r krb5kdc[27043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:19.569878 27125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:19.570519 27125 log.cc:824] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:19.572090 27125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64: No bootstrap required, opened a new log
I20260925 02:55:19.573750 27128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:19.562035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55712 (local address 127.12.222.254:34743)
0925 02:55:19.562439 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:19.562454 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:19.562485 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:55:19.563047 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:19.563062 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:19.563337 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:19.563586 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:19.564421 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:19.565337 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:19.566549 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:19.566979 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:19.567105 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:19.569694 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:19.569698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:19.569708 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:19.569731 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:19.571251 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:19.571676 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:19.571681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:19.571684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:19.571781 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:19.572055 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:19.572058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:19.572060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:19.572465 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:19.572647 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:19.572928 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 02:55:19.573165 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":123,"threads_started":1}
I20260925 02:55:19.575522 27125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00aaad83295f4139b7f27b616d20de64" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 34743 } }
I20260925 02:55:19.575802 27125 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:19.575882 27125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00aaad83295f4139b7f27b616d20de64, State: Initialized, Role: FOLLOWER
I20260925 02:55:19.576297 27125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00aaad83295f4139b7f27b616d20de64" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 34743 } }
I20260925 02:55:19.576414 27125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:19.576457 27125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:19.576522 27125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:19.577296 27125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00aaad83295f4139b7f27b616d20de64" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 34743 } }
I20260925 02:55:19.577565 27125 leader_election.cc:302] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00aaad83295f4139b7f27b616d20de64; no voters:
I20260925 02:55:19.577773 27125 leader_election.cc:288] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:19.577880 27131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:19.578097 27131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 1 LEADER]: Becoming Leader. State: Replica: 00aaad83295f4139b7f27b616d20de64, State: Running, Role: LEADER
I20260925 02:55:19.578374 27131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00aaad83295f4139b7f27b616d20de64" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 34743 } }
I20260925 02:55:19.578939 27125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:19.579505 27132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00aaad83295f4139b7f27b616d20de64. Latest consensus state: current_term: 1 leader_uuid: "00aaad83295f4139b7f27b616d20de64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00aaad83295f4139b7f27b616d20de64" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 34743 } } }
I20260925 02:55:19.579649 27132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:19.579880 27133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00aaad83295f4139b7f27b616d20de64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00aaad83295f4139b7f27b616d20de64" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 34743 } } }
I20260925 02:55:19.579998 27133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:19.580062 27138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:19.582528 27138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:19.587437 27138 catalog_manager.cc:1347] Generated new cluster ID: 7ba504fd0cb84d4aa63a47ff0bb6794f
I20260925 02:55:19.587494 27138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:19.595769 27138 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:19.596815 27138 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:19.608834 27138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64: Generated new TSK 0
I20260925 02:55:19.609479 27138 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
I20260925 02:55:19.640684 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:34743
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:19.746352 27152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:19.746554 27152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:19.746623 27152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:19.750270 27152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:19.750315 27152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:19.750422 27152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:55:19.755172 27152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:19.756037 27152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:19.757030 27152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:19.765535 27158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:19.765575 27152 server_base.cc:1034] running on GCE node
W20260925 02:55:19.765643 27160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:19.765535 27157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:19.766175 27152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:19.766747 27152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:19.767939 27152 hybrid_clock.cc:648] HybridClock initialized: now 1790304919767917 us; error 43 us; skew 500 ppm
Sep 25 02:55:19 dist-test-slave-252r krb5kdc[27043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304919, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:19.770428 27152 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:55:19.771130 27152 webserver.cc:460] Webserver started at http://127.12.222.193:32829/ using document root <none> and password file <none>
I20260925 02:55:19.771607 27152 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:19.771689 27152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:19.771886 27152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:19.773756 27152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "5bcc077b96d24bf8b37ced1d702717d3"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-252r"
server_key: "47c7c84771d59213bdf021b1b6f31f02"
server_key_iv: "7544f6af651365d9eea746bdaebeb4e6"
server_key_version: "encryptionkey@0"
I20260925 02:55:19.774309 27152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "5bcc077b96d24bf8b37ced1d702717d3"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-252r"
server_key: "47c7c84771d59213bdf021b1b6f31f02"
server_key_iv: "7544f6af651365d9eea746bdaebeb4e6"
server_key_version: "encryptionkey@0"
I20260925 02:55:19.777808 27152 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 02:55:19.780267 27167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:19.781464 27152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:55:19.781594 27152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "5bcc077b96d24bf8b37ced1d702717d3"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-252r"
server_key: "47c7c84771d59213bdf021b1b6f31f02"
server_key_iv: "7544f6af651365d9eea746bdaebeb4e6"
server_key_version: "encryptionkey@0"
I20260925 02:55:19.781695 27152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:19.800997 27152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:19.803895 27152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:19.804088 27152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:19.804664 27152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:19.805445 27152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:19.805490 27152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:19.805557 27152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:19.805653 27152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:19.813920 27152 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:40341
I20260925 02:55:19.814002 27282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:40341 every 8 connection(s)
I20260925 02:55:19.814779 27152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:55:19.815925 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 27152
I20260925 02:55:19.816080 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:55:19.816495 13179 external_mini_cluster.cc:1428] Setting key 6dede26d5bffb83997da0b9b9cd93528
Sep 25 02:55:19 dist-test-slave-252r krb5kdc[27043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304919, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.222.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.194@KRBTEST.COM" created.
I20260925 02:55:19.829947 27128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:19.816532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:53309 (local address 127.12.222.254:34743)
0925 02:55:19.816646 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:19.816651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:19.818149 (+ 1498us) server_negotiation.cc:407] Connection header received
0925 02:55:19.819424 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:19.819430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:19.819496 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:19.819586 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:19.821168 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:19.821682 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:19.823414 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:19.823625 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:19.823699 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:19.826332 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:19.826336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:19.826339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:19.826365 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:19.827712 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:19.828291 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:19.828294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:19.828296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:19.828368 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:19.828740 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:19.828744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:19.828746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:19.828964 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:19.829041 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:19.829542 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 02:55:19.829685 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:55:19.830456 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:19.816997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:53309)
0925 02:55:19.817981 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:19.818037 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:19.819184 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:19.819786 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:19.819795 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:19.820308 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:19.821013 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:19.821035 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:19.821824 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:19.821830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:19.823286 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:19.823299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:19.823773 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:19.823779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:19.823984 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:19.824529 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:19.824552 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:19.826167 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:19.827811 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:19.827815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:19.827825 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:19.828139 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:19.828454 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:19.828457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:19.828458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:19.828629 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:19.829069 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:19.829078 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:19.829299 (+ 221us) client_negotiation.cc:769] Sending connection context
0925 02:55:19.829459 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 02:55:19.829678 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":148,"threads_started":1}
I20260925 02:55:19.831698 27283 heartbeater.cc:347] Connected to a master server at 127.12.222.254:34743
I20260925 02:55:19.831918 27283 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:19.832505 27283 heartbeater.cc:511] Master 127.12.222.254:34743 requested a full tablet report, sending...
I20260925 02:55:19.833976 27089 ts_manager.cc:194] Registered new tserver with Master: 5bcc077b96d24bf8b37ced1d702717d3 (127.12.222.193:40341)
I20260925 02:55:19.835073 27089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:53309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
I20260925 02:55:19.843020 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:34743
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:19.942822 27288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:19.943027 27288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:19.943079 27288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:19.946704 27288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:19.946744 27288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:19.946796 27288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:55:19.951478 27288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:19.952291 27288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:19.953102 27288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:19.961539 27294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:19.961544 27293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:19.961581 27288 server_base.cc:1034] running on GCE node
W20260925 02:55:19.961539 27296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:19.962081 27288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:19.962595 27288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:19.963768 27288 hybrid_clock.cc:648] HybridClock initialized: now 1790304919963752 us; error 28 us; skew 500 ppm
Sep 25 02:55:19 dist-test-slave-252r krb5kdc[27043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304919, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:19.966214 27288 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:55:19.967010 27288 webserver.cc:460] Webserver started at http://127.12.222.194:34483/ using document root <none> and password file <none>
I20260925 02:55:19.967495 27288 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:19.967581 27288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:19.967780 27288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:19.969555 27288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "fed270065b044d3e99692d0663f3aab2"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-252r"
server_key: "8be58501d7b0276e50968a992ca140fa"
server_key_iv: "5c85abbff519cee9da3ad7c644563b0c"
server_key_version: "encryptionkey@0"
I20260925 02:55:19.970065 27288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "fed270065b044d3e99692d0663f3aab2"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-252r"
server_key: "8be58501d7b0276e50968a992ca140fa"
server_key_iv: "5c85abbff519cee9da3ad7c644563b0c"
server_key_version: "encryptionkey@0"
I20260925 02:55:19.973420 27288 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:55:19.975523 27303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:19.976696 27288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:55:19.976843 27288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "fed270065b044d3e99692d0663f3aab2"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-252r"
server_key: "8be58501d7b0276e50968a992ca140fa"
server_key_iv: "5c85abbff519cee9da3ad7c644563b0c"
server_key_version: "encryptionkey@0"
I20260925 02:55:19.976954 27288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:19.987830 27288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:19.990777 27288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:19.990953 27288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:19.991437 27288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:19.992146 27288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:19.992214 27288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:19.992313 27288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:19.992357 27288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:20.000771 27288 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:40313
I20260925 02:55:20.000814 27416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:40313 every 8 connection(s)
I20260925 02:55:20.001543 27288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:55:20.007961 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 27288
I20260925 02:55:20.008105 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:55:20.008438 13179 external_mini_cluster.cc:1428] Setting key a1cfaf2bfd9a0d447abca0b3068b6ad0
Sep 25 02:55:20 dist-test-slave-252r krb5kdc[27043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304919, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:20.014384 27128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.003188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:47679 (local address 127.12.222.254:34743)
0925 02:55:20.003333 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:20.003337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:20.004072 (+ 735us) server_negotiation.cc:407] Connection header received
0925 02:55:20.005269 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:20.005271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:20.005318 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:20.005388 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:20.006359 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.006858 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.007900 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.008212 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.008334 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.010725 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:20.010728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:20.010731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:20.010765 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:20.012311 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:20.012880 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:20.012882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:20.012883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:20.012964 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:20.013327 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:20.013329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:20.013330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:20.013493 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:20.013565 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:20.014175 (+ 610us) server_negotiation.cc:299] Negotiation successful
0925 02:55:20.014272 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:55:20.015570 27419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.003512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:47679)
0925 02:55:20.003917 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:20.003955 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:20.005089 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:20.005503 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:20.005509 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:20.005814 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:20.006238 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.006249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.006966 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.006968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.007764 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.007772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.008368 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.008372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.008578 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.009169 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:20.009183 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:20.010568 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:20.012418 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:20.012427 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:20.012443 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:20.012778 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:20.013049 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:20.013055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:20.013060 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:20.013236 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:20.013598 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:20.013607 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:20.013901 (+ 294us) client_negotiation.cc:769] Sending connection context
0925 02:55:20.014134 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 02:55:20.014456 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":116,"threads_started":1}
I20260925 02:55:20.016760 27417 heartbeater.cc:347] Connected to a master server at 127.12.222.254:34743
I20260925 02:55:20.017011 27417 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:20.017558 27417 heartbeater.cc:511] Master 127.12.222.254:34743 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.222.195@KRBTEST.COM; defaulting to no policy
I20260925 02:55:20.018558 27089 ts_manager.cc:194] Registered new tserver with Master: fed270065b044d3e99692d0663f3aab2 (127.12.222.194:40313)
I20260925 02:55:20.019052 27089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:47679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
I20260925 02:55:20.033357 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:34743
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:20.134243 27422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:20.134431 27422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:20.134465 27422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:20.138067 27422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:20.138110 27422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:20.138163 27422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:55:20.142656 27422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:20.143380 27422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:20.144160 27422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:20.152827 27430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:20.152839 27428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:20.152843 27427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:20.153157 27422 server_base.cc:1034] running on GCE node
I20260925 02:55:20.153558 27422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:20.154098 27422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:20.155272 27422 hybrid_clock.cc:648] HybridClock initialized: now 1790304920155265 us; error 26 us; skew 500 ppm
Sep 25 02:55:20 dist-test-slave-252r krb5kdc[27043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304920, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:20.157820 27422 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:55:20.158560 27422 webserver.cc:460] Webserver started at http://127.12.222.195:42575/ using document root <none> and password file <none>
I20260925 02:55:20.159075 27422 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:20.159158 27422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:20.159354 27422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:20.161166 27422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "3473197b94e8476f8143bb534ada9f09"
format_stamp: "Formatted at 2026-09-25 02:55:20 on dist-test-slave-252r"
server_key: "e51924c6a7472509cdc56e646669b032"
server_key_iv: "22382cda677f14071534882fe70a3506"
server_key_version: "encryptionkey@0"
I20260925 02:55:20.161692 27422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "3473197b94e8476f8143bb534ada9f09"
format_stamp: "Formatted at 2026-09-25 02:55:20 on dist-test-slave-252r"
server_key: "e51924c6a7472509cdc56e646669b032"
server_key_iv: "22382cda677f14071534882fe70a3506"
server_key_version: "encryptionkey@0"
I20260925 02:55:20.164943 27422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:55:20.167130 27438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:20.168274 27422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:55:20.168396 27422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "3473197b94e8476f8143bb534ada9f09"
format_stamp: "Formatted at 2026-09-25 02:55:20 on dist-test-slave-252r"
server_key: "e51924c6a7472509cdc56e646669b032"
server_key_iv: "22382cda677f14071534882fe70a3506"
server_key_version: "encryptionkey@0"
I20260925 02:55:20.168488 27422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:20.187914 27422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:20.190382 27422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:20.190515 27422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:20.190958 27422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:20.191632 27422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:20.191674 27422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:20.191709 27422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:20.191725 27422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:20.200453 27422 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:40297
I20260925 02:55:20.200484 27551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:40297 every 8 connection(s)
I20260925 02:55:20.201284 27422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:55:20.208468 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 27422
I20260925 02:55:20.208603 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:55:20.208912 13179 external_mini_cluster.cc:1428] Setting key cf330eec8d6d0f23e7ef444e4c439a18
Sep 25 02:55:20 dist-test-slave-252r krb5kdc[27043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304920, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:20.213928 27128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.202858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:43345 (local address 127.12.222.254:34743)
0925 02:55:20.202990 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:20.202993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:20.203797 (+ 804us) server_negotiation.cc:407] Connection header received
0925 02:55:20.205069 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:20.205072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:20.205141 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:20.205213 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:20.206218 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.206730 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.207768 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.207993 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.208098 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.210554 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:20.210557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:20.210560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:20.210585 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:20.211979 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:20.212516 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:20.212519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:20.212520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:20.212596 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:20.212916 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:20.212919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:20.212921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:20.213083 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:20.213201 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:20.213651 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 02:55:20.213760 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:55:20.214564 27554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.203231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:43345)
0925 02:55:20.203654 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:20.203693 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:20.204859 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:20.205359 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:20.205365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:20.205667 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:20.206080 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.206092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.206852 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.206857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.207646 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.207654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.208132 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.208136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.208376 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.208925 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:20.208946 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:20.210408 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:20.212108 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:20.212113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:20.212124 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:20.212422 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:20.212683 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:20.212687 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:20.212689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:20.212815 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:20.213200 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:20.213206 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:20.213427 (+ 221us) client_negotiation.cc:769] Sending connection context
0925 02:55:20.213593 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 02:55:20.213867 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":117,"threads_started":1}
I20260925 02:55:20.215466 27552 heartbeater.cc:347] Connected to a master server at 127.12.222.254:34743
I20260925 02:55:20.215678 27552 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:20.216126 27552 heartbeater.cc:511] Master 127.12.222.254:34743 requested a full tablet report, sending...
I20260925 02:55:20.217072 27089 ts_manager.cc:194] Registered new tserver with Master: 3473197b94e8476f8143bb534ada9f09 (127.12.222.195:40297)
I20260925 02:55:20.217609 27089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:43345
I20260925 02:55:20.222442 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:55:20.231177 27128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.224231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55726 (local address 127.12.222.254:34743)
0925 02:55:20.224379 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:20.224382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:20.224626 (+ 244us) server_negotiation.cc:407] Connection header received
0925 02:55:20.224771 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:20.224775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:20.224828 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:20.224905 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:20.225549 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.226024 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.226830 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.227096 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.227216 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.228457 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:20.228460 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:20.228462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:20.228485 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:20.229619 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:20.230027 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:20.230030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:20.230031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:20.230108 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:20.230361 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:20.230364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:20.230366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:20.230579 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:20.230725 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:20.230936 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 02:55:20.231031 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 02:55:20.233593 27089 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:55:20.236207 27089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:55:20.249409 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.244349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42018 (local address 127.12.222.193:40341)
0925 02:55:20.245069 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:20.245078 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:20.245099 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:55:20.245168 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:20.245176 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:20.245364 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:20.245505 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:20.245905 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.246866 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.248175 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.248792 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.248925 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.249035 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:20.249193 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 02:55:20.249274 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":102,"threads_started":1}
I20260925 02:55:20.249410 27563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.243703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:40341 (local address 127.0.0.1:42018)
0925 02:55:20.244871 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:20.244885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:20.244978 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:20.245667 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:20.245670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:20.245686 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:20.245775 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.245780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.247061 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.247064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.248056 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.248066 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.248909 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.248913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.249010 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.249044 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 02:55:20.249166 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 02:55:20.249274 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":64,"spinlock_wait_cycles":12928,"thread_start_us":86,"threads_started":1}
I20260925 02:55:20.249776 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.244326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52096 (local address 127.12.222.195:40297)
0925 02:55:20.244795 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:20.244807 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:20.244829 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:55:20.244900 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:20.244906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:20.245119 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:20.245281 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:20.245497 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.246486 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.248044 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.248798 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.248905 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.249012 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:20.249553 (+ 541us) server_negotiation.cc:299] Negotiation successful
0925 02:55:20.249651 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":134,"threads_started":1}
I20260925 02:55:20.249882 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.243625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:40313 (local address 127.0.0.1:56432)
0925 02:55:20.244299 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:20.244340 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:20.244454 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:20.246006 (+ 1552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:20.246009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:20.246026 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:20.246111 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.246117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.247337 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.247339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.248384 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.248402 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.249509 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.249512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.249591 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.249610 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 02:55:20.249682 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 02:55:20.249737 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260925 02:55:20.250020 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.243738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.12.222.194:40313)
0925 02:55:20.245581 (+ 1843us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:20.245591 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:20.245610 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:55:20.245681 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:20.245685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:20.245869 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:20.246032 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:20.246203 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.247197 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.248516 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.249250 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.249347 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.249445 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:20.249732 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 02:55:20.249835 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1736,"thread_start_us":93,"threads_started":1}
I20260925 02:55:20.249467 27564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.243960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:40297 (local address 127.0.0.1:52096)
0925 02:55:20.244706 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:20.244719 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:20.244801 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:20.245277 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:20.245282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:20.245298 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:20.245383 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.245387 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.246628 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.246631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.247922 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.247932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.249078 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.249081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.249162 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.249184 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:55:20.249256 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 02:55:20.249334 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":38,"thread_start_us":81,"threads_started":1}
I20260925 02:55:20.252390 27216 tablet_service.cc:1467] Processing CreateTablet for tablet 01200e2b1a9a42c2919e20d1b5c39f93 (DEFAULT_TABLE table=test-table [id=5f62b8fdc11c49e99ac890f0d954eb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:20.252394 27351 tablet_service.cc:1467] Processing CreateTablet for tablet 01200e2b1a9a42c2919e20d1b5c39f93 (DEFAULT_TABLE table=test-table [id=5f62b8fdc11c49e99ac890f0d954eb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:20.252892 27486 tablet_service.cc:1467] Processing CreateTablet for tablet 01200e2b1a9a42c2919e20d1b5c39f93 (DEFAULT_TABLE table=test-table [id=5f62b8fdc11c49e99ac890f0d954eb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:20.253340 27216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01200e2b1a9a42c2919e20d1b5c39f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:20.253399 27351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01200e2b1a9a42c2919e20d1b5c39f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:20.253779 27486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01200e2b1a9a42c2919e20d1b5c39f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:20.258991 27569 tablet_bootstrap.cc:492] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3: Bootstrap starting.
I20260925 02:55:20.260417 27570 tablet_bootstrap.cc:492] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09: Bootstrap starting.
I20260925 02:55:20.260695 27571 tablet_bootstrap.cc:492] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2: Bootstrap starting.
I20260925 02:55:20.261698 27569 tablet_bootstrap.cc:654] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:20.262356 27570 tablet_bootstrap.cc:654] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:20.262349 27569 log.cc:824] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:20.262812 27571 tablet_bootstrap.cc:654] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:20.263574 27571 log.cc:824] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:20.263574 27570 log.cc:824] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:20.264060 27569 tablet_bootstrap.cc:492] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3: No bootstrap required, opened a new log
I20260925 02:55:20.264281 27569 ts_tablet_manager.cc:1397] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 02:55:20.265496 27570 tablet_bootstrap.cc:492] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09: No bootstrap required, opened a new log
I20260925 02:55:20.265642 27571 tablet_bootstrap.cc:492] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2: No bootstrap required, opened a new log
I20260925 02:55:20.265662 27570 ts_tablet_manager.cc:1397] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:55:20.265789 27571 ts_tablet_manager.cc:1397] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:55:20.267529 27569 raft_consensus.cc:348] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } } peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } }
I20260925 02:55:20.267745 27569 raft_consensus.cc:374] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:20.267797 27569 raft_consensus.cc:729] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bcc077b96d24bf8b37ced1d702717d3, State: Initialized, Role: FOLLOWER
I20260925 02:55:20.268272 27569 consensus_queue.cc:260] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } } peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } }
I20260925 02:55:20.268879 27283 heartbeater.cc:503] Master 127.12.222.254:34743 was elected leader, sending a full tablet report...
I20260925 02:55:20.269222 27569 ts_tablet_manager.cc:1428] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:55:20.269323 27571 raft_consensus.cc:348] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } } peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } }
I20260925 02:55:20.269323 27570 raft_consensus.cc:348] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } } peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } }
I20260925 02:55:20.269567 27571 raft_consensus.cc:374] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:20.269565 27570 raft_consensus.cc:374] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:20.269634 27570 raft_consensus.cc:729] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3473197b94e8476f8143bb534ada9f09, State: Initialized, Role: FOLLOWER
I20260925 02:55:20.269634 27571 raft_consensus.cc:729] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed270065b044d3e99692d0663f3aab2, State: Initialized, Role: FOLLOWER
I20260925 02:55:20.270040 27570 consensus_queue.cc:260] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } } peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } }
I20260925 02:55:20.270040 27571 consensus_queue.cc:260] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } } peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } }
I20260925 02:55:20.270828 27552 heartbeater.cc:503] Master 127.12.222.254:34743 was elected leader, sending a full tablet report...
I20260925 02:55:20.270893 27570 ts_tablet_manager.cc:1428] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 02:55:20.270893 27571 ts_tablet_manager.cc:1428] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 02:55:20.271799 27417 heartbeater.cc:503] Master 127.12.222.254:34743 was elected leader, sending a full tablet report...
I20260925 02:55:20.285403 27578 raft_consensus.cc:482] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:55:20.285543 27578 raft_consensus.cc:504] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } } peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } }
I20260925 02:55:20.286481 27578 leader_election.cc:288] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bcc077b96d24bf8b37ced1d702717d3 (127.12.222.193:40341), 3473197b94e8476f8143bb534ada9f09 (127.12.222.195:40297)
I20260925 02:55:20.290360 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.286761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:42491 (local address 127.12.222.195:40297)
0925 02:55:20.286876 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:20.286880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:20.287051 (+ 171us) server_negotiation.cc:407] Connection header received
0925 02:55:20.287123 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:20.287128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:20.287176 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:20.287240 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:20.287493 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.288269 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.289391 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.289791 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.289907 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.289960 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:20.290181 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 02:55:20.290249 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:55:20.290369 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.286699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:33283 (local address 127.12.222.193:40341)
0925 02:55:20.286803 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:20.286807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:20.286917 (+ 110us) server_negotiation.cc:407] Connection header received
0925 02:55:20.287043 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:20.287046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:20.287094 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:20.287176 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:20.287491 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.288244 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.289388 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.289789 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.289908 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.289954 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:20.290213 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 02:55:20.290279 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:55:20.290361 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.286674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:40297 (local address 127.12.222.194:42491)
0925 02:55:20.286961 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:20.286976 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:20.287061 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:20.287266 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:20.287269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:20.287280 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:20.287380 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.287388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.288404 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.288409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.289267 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.289277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.289909 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.289912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.290030 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.290059 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 02:55:20.290132 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:55:20.290211 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":57,"thread_start_us":109,"threads_started":1}
I20260925 02:55:20.290362 27419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.286660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:40341 (local address 127.12.222.194:33283)
0925 02:55:20.286825 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:20.286840 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:20.286928 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:20.287188 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:20.287190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:20.287203 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:20.287338 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.287359 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.288387 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.288392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.289265 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.289274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.289901 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.289904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.290054 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.290070 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:55:20.290182 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:55:20.290248 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":64}
I20260925 02:55:20.290906 27506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01200e2b1a9a42c2919e20d1b5c39f93" candidate_uuid: "fed270065b044d3e99692d0663f3aab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3473197b94e8476f8143bb534ada9f09" is_pre_election: true
I20260925 02:55:20.291131 27236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01200e2b1a9a42c2919e20d1b5c39f93" candidate_uuid: "fed270065b044d3e99692d0663f3aab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bcc077b96d24bf8b37ced1d702717d3" is_pre_election: true
I20260925 02:55:20.291164 27506 raft_consensus.cc:2454] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed270065b044d3e99692d0663f3aab2 in term 0.
I20260925 02:55:20.291378 27236 raft_consensus.cc:2454] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed270065b044d3e99692d0663f3aab2 in term 0.
I20260925 02:55:20.291630 27307 leader_election.cc:302] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3473197b94e8476f8143bb534ada9f09, fed270065b044d3e99692d0663f3aab2; no voters:
I20260925 02:55:20.291883 27578 raft_consensus.cc:2781] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:55:20.291968 27578 raft_consensus.cc:482] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:55:20.292045 27578 raft_consensus.cc:3037] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:20.293035 27578 raft_consensus.cc:504] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } } peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } }
I20260925 02:55:20.293321 27578 leader_election.cc:288] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [CANDIDATE]: Term 1 election: Requested vote from peers 5bcc077b96d24bf8b37ced1d702717d3 (127.12.222.193:40341), 3473197b94e8476f8143bb534ada9f09 (127.12.222.195:40297)
I20260925 02:55:20.293622 27236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01200e2b1a9a42c2919e20d1b5c39f93" candidate_uuid: "fed270065b044d3e99692d0663f3aab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bcc077b96d24bf8b37ced1d702717d3"
I20260925 02:55:20.293671 27506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01200e2b1a9a42c2919e20d1b5c39f93" candidate_uuid: "fed270065b044d3e99692d0663f3aab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3473197b94e8476f8143bb534ada9f09"
I20260925 02:55:20.293754 27236 raft_consensus.cc:3037] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:20.293803 27506 raft_consensus.cc:3037] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:20.294699 27236 raft_consensus.cc:2454] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed270065b044d3e99692d0663f3aab2 in term 1.
I20260925 02:55:20.294700 27506 raft_consensus.cc:2454] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed270065b044d3e99692d0663f3aab2 in term 1.
I20260925 02:55:20.294965 27306 leader_election.cc:302] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bcc077b96d24bf8b37ced1d702717d3, fed270065b044d3e99692d0663f3aab2; no voters:
I20260925 02:55:20.295140 27578 raft_consensus.cc:2781] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:20.295349 27578 raft_consensus.cc:686] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [term 1 LEADER]: Becoming Leader. State: Replica: fed270065b044d3e99692d0663f3aab2, State: Running, Role: LEADER
I20260925 02:55:20.295632 27578 consensus_queue.cc:237] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } } peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } }
I20260925 02:55:20.298591 27086 catalog_manager.cc:5495] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 reported cstate change: term changed from 0 to 1, leader changed from <none> to fed270065b044d3e99692d0663f3aab2 (127.12.222.194). New cstate: current_term: 1 leader_uuid: "fed270065b044d3e99692d0663f3aab2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } health_report { overall_health: UNKNOWN } } }
I20260925 02:55:20.311333 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.308486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56446 (local address 127.12.222.194:40313)
0925 02:55:20.308600 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:20.308604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:20.308615 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:55:20.308730 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:20.308732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:20.308773 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:20.308852 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:55:20.309055 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.309487 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.310220 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.310403 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.310471 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.310730 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:55:20.311072 (+ 342us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:55:20.311154 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:20.311207 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 02:55:20.311248 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 02:55:20.315824 27284 tablet.cc:2369] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:55:20.316144 27236 raft_consensus.cc:1261] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Refusing update from remote peer fed270065b044d3e99692d0663f3aab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:20.316143 27506 raft_consensus.cc:1261] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09 [term 1 FOLLOWER]: Refusing update from remote peer fed270065b044d3e99692d0663f3aab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:20.316843 27578 consensus_queue.cc:1035] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:20.317049 27580 consensus_queue.cc:1035] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:20.326049 27584 mvcc.cc:205] Tried to move back new op lower bound from 7333088953609113600 to 7333088953533636608. Current Snapshot: MvccSnapshot[applied={T|T < 7333088953609113600}]
I20260925 02:55:20.330549 27086 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:20.330670 27086 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:20.332897 27086 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64: Sending DeleteTablet for 3 replicas of tablet 01200e2b1a9a42c2919e20d1b5c39f93
I20260925 02:55:20.333554 27351 tablet_service.cc:1514] Processing DeleteTablet for tablet 01200e2b1a9a42c2919e20d1b5c39f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:20 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:56432
I20260925 02:55:20.333551 27216 tablet_service.cc:1514] Processing DeleteTablet for tablet 01200e2b1a9a42c2919e20d1b5c39f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:20 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:42018
I20260925 02:55:20.333639 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 27152
I20260925 02:55:20.334851 27486 tablet_service.cc:1514] Processing DeleteTablet for tablet 01200e2b1a9a42c2919e20d1b5c39f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:20 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:52096
I20260925 02:55:20.334928 27592 tablet_replica.cc:330] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2: stopping tablet replica
I20260925 02:55:20.335202 27592 raft_consensus.cc:2229] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:55:20.335512 27592 raft_consensus.cc:2258] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:20.335845 27593 tablet_replica.cc:330] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09: stopping tablet replica
I20260925 02:55:20.336162 27593 raft_consensus.cc:2229] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:20.336511 27593 raft_consensus.cc:2258] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:20.336974 27592 ts_tablet_manager.cc:1905] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:20.339044 27592 ts_tablet_manager.cc:1918] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:20.339145 27592 log.cc:1196] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/01200e2b1a9a42c2919e20d1b5c39f93
I20260925 02:55:20.339509 27592 ts_tablet_manager.cc:1939] T 01200e2b1a9a42c2919e20d1b5c39f93 P fed270065b044d3e99692d0663f3aab2: Deleting consensus metadata
I20260925 02:55:20.340302 27593 ts_tablet_manager.cc:1905] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:20.340611 27073 catalog_manager.cc:4841] TS fed270065b044d3e99692d0663f3aab2 (127.12.222.194:40313): tablet 01200e2b1a9a42c2919e20d1b5c39f93 (table test-table [id=5f62b8fdc11c49e99ac890f0d954eb68]) successfully deleted
W20260925 02:55:20.342242 27075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.222.193:40341 (error 108)
I20260925 02:55:20.342417 27593 ts_tablet_manager.cc:1918] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:20.342523 27593 log.cc:1196] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/01200e2b1a9a42c2919e20d1b5c39f93
I20260925 02:55:20.342803 27593 ts_tablet_manager.cc:1939] T 01200e2b1a9a42c2919e20d1b5c39f93 P 3473197b94e8476f8143bb534ada9f09: Deleting consensus metadata
I20260925 02:55:20.342957 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 27288
I20260925 02:55:20.344452 27564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.344129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:40341 (local address 127.0.0.1:42032)
0925 02:55:20.344303 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:20.344366 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.193:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260925 02:55:20.344595 27075 catalog_manager.cc:4584] TS 5bcc077b96d24bf8b37ced1d702717d3 (127.12.222.193:40341): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01200e2b1a9a42c2919e20d1b5c39f93: Network error: Client connection negotiation failed: client connection to 127.12.222.193:40341: connect: Connection refused (error 111)
I20260925 02:55:20.345052 27076 catalog_manager.cc:4841] TS 3473197b94e8476f8143bb534ada9f09 (127.12.222.195:40297): tablet 01200e2b1a9a42c2919e20d1b5c39f93 (table test-table [id=5f62b8fdc11c49e99ac890f0d954eb68]) successfully deleted
I20260925 02:55:20.348793 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 27422
I20260925 02:55:20.354295 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 27056
I20260925 02:55:20.360095 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:34743
--webserver_interface=127.12.222.254
--webserver_port=37109
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:34743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:20.459048 27595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:20.459264 27595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:20.459312 27595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:20.463012 27595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:20.463058 27595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:20.463076 27595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:20.463091 27595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:20.463104 27595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:20.467667 27595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:34743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:34743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=37109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:20.468554 27595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:20.469383 27595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:20.481812 27601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:20.481813 27602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:20.482041 27595 server_base.cc:1034] running on GCE node
W20260925 02:55:20.481842 27604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:20.482748 27595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:20.483763 27595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:20.484943 27595 hybrid_clock.cc:648] HybridClock initialized: now 1790304920484924 us; error 31 us; skew 500 ppm
Sep 25 02:55:20 dist-test-slave-252r krb5kdc[27043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304920, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:20.487501 27595 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:55:20.488411 27595 webserver.cc:460] Webserver started at http://127.12.222.254:37109/ using document root <none> and password file <none>
I20260925 02:55:20.488956 27595 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:20.489059 27595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:20.492520 27595 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 02:55:20.494431 27611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:20.495527 27595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:55:20.495661 27595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "00aaad83295f4139b7f27b616d20de64"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-252r"
server_key: "7b8015f0f1e449df50c207b4ed7b0f48"
server_key_iv: "ba18287f4dce18c78d499b5154978176"
server_key_version: "encryptionkey@0"
I20260925 02:55:20.496212 27595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:20.508284 27595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:20.511116 27595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:20.511291 27595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:20.518350 27595 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:34743
I20260925 02:55:20.518666 27663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:34743 every 8 connection(s)
I20260925 02:55:20.519430 27595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:20.523506 27664 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:55:20.525085 27664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64: Bootstrap starting.
I20260925 02:55:20.525569 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 27595
I20260925 02:55:20.526072 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:40341
--local_ip_for_outbound_sockets=127.12.222.193
--tserver_master_addrs=127.12.222.254:34743
--webserver_port=32829
--webserver_interface=127.12.222.193
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:55:20.542357 27664 log.cc:824] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:20.550352 27664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:55:20.550746 27664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64: Bootstrap complete.
I20260925 02:55:20.554585 27664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00aaad83295f4139b7f27b616d20de64" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 34743 } }
I20260925 02:55:20.554967 27664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00aaad83295f4139b7f27b616d20de64, State: Initialized, Role: FOLLOWER
I20260925 02:55:20.555390 27664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00aaad83295f4139b7f27b616d20de64" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 34743 } }
I20260925 02:55:20.555511 27664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:20.555598 27664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:20.555706 27664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:55:20.557495 27664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00aaad83295f4139b7f27b616d20de64" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 34743 } }
I20260925 02:55:20.557803 27664 leader_election.cc:302] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00aaad83295f4139b7f27b616d20de64; no voters:
I20260925 02:55:20.558066 27664 leader_election.cc:288] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:55:20.558183 27668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:55:20.558458 27668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 2 LEADER]: Becoming Leader. State: Replica: 00aaad83295f4139b7f27b616d20de64, State: Running, Role: LEADER
I20260925 02:55:20.558751 27668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00aaad83295f4139b7f27b616d20de64" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 34743 } }
I20260925 02:55:20.559254 27664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:20.560120 27670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00aaad83295f4139b7f27b616d20de64. Latest consensus state: current_term: 2 leader_uuid: "00aaad83295f4139b7f27b616d20de64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00aaad83295f4139b7f27b616d20de64" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 34743 } } }
I20260925 02:55:20.560285 27670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:20.560607 27669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "00aaad83295f4139b7f27b616d20de64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00aaad83295f4139b7f27b616d20de64" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 34743 } } }
I20260925 02:55:20.560720 27669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:20.560608 27677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:20.565024 27677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:20.565567 27677 catalog_manager.cc:1259] Loaded cluster ID: 7ba504fd0cb84d4aa63a47ff0bb6794f
I20260925 02:55:20.565621 27677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:20.566855 27677 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 02:55:20.571306 27677 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 02:55:20.575953 27677 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:20.576642 27677 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64: Loaded TSK: 0
I20260925 02:55:20.577757 27677 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:55:20.634258 27666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:20.634506 27666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:20.634605 27666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:20.638393 27666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:20.638463 27666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:20.638535 27666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:55:20.643424 27666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:40341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=32829
--tserver_master_addrs=127.12.222.254:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:20.644309 27666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:20.645196 27666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:20.654971 27692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:20.654945 27694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:20.654947 27691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:20.655457 27666 server_base.cc:1034] running on GCE node
I20260925 02:55:20.655820 27666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:20.656546 27666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:20.657717 27666 hybrid_clock.cc:648] HybridClock initialized: now 1790304920657692 us; error 42 us; skew 500 ppm
Sep 25 02:55:20 dist-test-slave-252r krb5kdc[27043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304920, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:20.660502 27666 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:55:20.661381 27666 webserver.cc:460] Webserver started at http://127.12.222.193:32829/ using document root <none> and password file <none>
I20260925 02:55:20.661923 27666 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:20.662029 27666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:20.665674 27666 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:55:20.667785 27701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:20.669018 27666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:55:20.669149 27666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "5bcc077b96d24bf8b37ced1d702717d3"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-252r"
server_key: "47c7c84771d59213bdf021b1b6f31f02"
server_key_iv: "7544f6af651365d9eea746bdaebeb4e6"
server_key_version: "encryptionkey@0"
I20260925 02:55:20.669672 27666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:20.686581 27666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:20.689648 27666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:20.689841 27666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:20.690459 27666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:20.691484 27708 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 02:55:20.695336 27666 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:55:20.695405 27666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260925 02:55:20.695447 27666 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:55:20.697162 27666 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:55:20.697229 27666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260925 02:55:20.697458 27708 tablet_bootstrap.cc:492] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3: Bootstrap starting.
I20260925 02:55:20.707257 27666 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:40341
I20260925 02:55:20.707355 27815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:40341 every 8 connection(s)
I20260925 02:55:20.708107 27666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:55:20.712288 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 27666
I20260925 02:55:20.712742 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:40313
--local_ip_for_outbound_sockets=127.12.222.194
--tserver_master_addrs=127.12.222.254:34743
--webserver_port=34483
--webserver_interface=127.12.222.194
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:55:20.716600 27708 log.cc:824] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:55:20 dist-test-slave-252r krb5kdc[27043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304920, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:20.724457 27708 tablet_bootstrap.cc:492] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:55:20.724879 27708 tablet_bootstrap.cc:492] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3: Bootstrap complete.
I20260925 02:55:20.726152 27708 ts_tablet_manager.cc:1397] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3: Time spent bootstrapping tablet: real 0.029s user 0.018s sys 0.008s
I20260925 02:55:20.726392 27819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.709889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:54281)
0925 02:55:20.710287 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:20.710333 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:20.711038 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:20.712417 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:20.712425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:20.712844 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:20.713432 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.713449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.715141 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.715147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.716596 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.716611 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.717757 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.717760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.717940 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.718586 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:20.718604 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:20.720567 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:20.722985 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:20.722993 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:20.723009 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:20.723420 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:20.723834 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:20.723837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:20.723839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:20.723997 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:20.724713 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:20.724722 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:20.724993 (+ 271us) client_negotiation.cc:769] Sending connection context
0925 02:55:20.725166 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 02:55:20.725476 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":126,"threads_started":1}
I20260925 02:55:20.726418 27818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.709916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:54281 (local address 127.12.222.254:34743)
0925 02:55:20.710343 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:20.710359 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:20.710450 (+ 91us) server_negotiation.cc:407] Connection header received
0925 02:55:20.711460 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:20.711475 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:20.711857 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:20.712303 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:20.713622 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.714971 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.716754 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.717606 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.717811 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.720803 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:20.720810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:20.720823 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:20.720854 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:20.722847 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:20.723564 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:20.723570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:20.723575 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:20.723718 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:20.724116 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:20.724120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:20.724122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:20.724594 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:20.724717 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:20.725305 (+ 588us) server_negotiation.cc:299] Negotiation successful
0925 02:55:20.725570 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":110,"threads_started":1}
I20260925 02:55:20.729069 27816 heartbeater.cc:347] Connected to a master server at 127.12.222.254:34743
I20260925 02:55:20.728710 27708 raft_consensus.cc:348] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } } peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } }
I20260925 02:55:20.729240 27708 raft_consensus.cc:729] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bcc077b96d24bf8b37ced1d702717d3, State: Initialized, Role: FOLLOWER
I20260925 02:55:20.729329 27816 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:20.729677 27708 consensus_queue.cc:260] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } } peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } }
I20260925 02:55:20.729789 27816 heartbeater.cc:511] Master 127.12.222.254:34743 requested a full tablet report, sending...
I20260925 02:55:20.730577 27708 ts_tablet_manager.cc:1428] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260925 02:55:20.732142 27628 ts_manager.cc:194] Registered new tserver with Master: 5bcc077b96d24bf8b37ced1d702717d3 (127.12.222.193:40341)
I20260925 02:55:20.735783 27628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:54281
I20260925 02:55:20.737906 27816 heartbeater.cc:503] Master 127.12.222.254:34743 was elected leader, sending a full tablet report...
Sep 25 02:55:20 dist-test-slave-252r krb5kdc[27043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304920, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for kudu/127.12.222.193@KRBTEST.COM
I20260925 02:55:20.744761 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.735218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:40341 (local address 127.0.0.1:42034)
0925 02:55:20.735687 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:20.735727 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:20.735855 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:20.736391 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:20.736396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:20.736750 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:20.736870 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.736878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.738165 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.738168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.738742 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.738751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.739491 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.739494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.739566 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.740003 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:20.740013 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:20.741473 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:20.743111 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:20.743120 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:20.743125 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:20.743327 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:20.743639 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:20.743642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:20.743644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:20.743727 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:20.744149 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:20.744155 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:20.744305 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 02:55:20.744444 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 02:55:20.744609 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":97,"threads_started":1}
I20260925 02:55:20.744792 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.735266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.12.222.193:40341)
0925 02:55:20.735930 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:20.735941 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:20.735963 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:55:20.736051 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:20.736055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:20.736200 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:20.736391 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:20.737007 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.738014 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.738868 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.739258 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.739354 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.741606 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:20.741615 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:20.741625 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:20.741662 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:20.743002 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:20.743457 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:20.743462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:20.743464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:20.743545 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:20.743816 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:20.743820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:20.743822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:20.744027 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:20.744172 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:20.744529 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 02:55:20.744665 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":93,"threads_started":1}
I20260925 02:55:20.745585 27750 tablet_service.cc:1514] Processing DeleteTablet for tablet 01200e2b1a9a42c2919e20d1b5c39f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:20 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:42034
I20260925 02:55:20.746388 27825 tablet_replica.cc:330] stopping tablet replica
I20260925 02:55:20.746649 27825 raft_consensus.cc:2229] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:20.746871 27825 raft_consensus.cc:2258] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:20.747670 27825 ts_tablet_manager.cc:1905] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:20.750923 27825 ts_tablet_manager.cc:1918] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:20.751029 27825 log.cc:1196] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/wal/wals/01200e2b1a9a42c2919e20d1b5c39f93
I20260925 02:55:20.751293 27825 ts_tablet_manager.cc:1939] T 01200e2b1a9a42c2919e20d1b5c39f93 P 5bcc077b96d24bf8b37ced1d702717d3: Deleting consensus metadata
I20260925 02:55:20.752727 27614 catalog_manager.cc:4841] TS 5bcc077b96d24bf8b37ced1d702717d3 (127.12.222.193:40341): tablet 01200e2b1a9a42c2919e20d1b5c39f93 (table test-table [id=5f62b8fdc11c49e99ac890f0d954eb68]) successfully deleted
W20260925 02:55:20.824710 27820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:20.824935 27820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:20.824973 27820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:20.828881 27820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:20.828931 27820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:20.829022 27820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:55:20.833897 27820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:40313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=34483
--tserver_master_addrs=127.12.222.254:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:20.834765 27820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:20.835657 27820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:20.844529 27830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:20.844679 27820 server_base.cc:1034] running on GCE node
W20260925 02:55:20.844599 27833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:20.844527 27831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:20.845135 27820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:20.845723 27820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:20.846903 27820 hybrid_clock.cc:648] HybridClock initialized: now 1790304920846878 us; error 49 us; skew 500 ppm
Sep 25 02:55:20 dist-test-slave-252r krb5kdc[27043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304920, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:20.849431 27820 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:55:20.850208 27820 webserver.cc:460] Webserver started at http://127.12.222.194:34483/ using document root <none> and password file <none>
I20260925 02:55:20.850723 27820 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:20.850831 27820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:20.854316 27820 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:55:20.856173 27840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:20.857329 27820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:55:20.857486 27820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "fed270065b044d3e99692d0663f3aab2"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-252r"
server_key: "8be58501d7b0276e50968a992ca140fa"
server_key_iv: "5c85abbff519cee9da3ad7c644563b0c"
server_key_version: "encryptionkey@0"
I20260925 02:55:20.857982 27820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:20.873248 27820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:20.875680 27820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:20.875857 27820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:20.876461 27820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:20.877205 27820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:20.877283 27820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:20.877344 27820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:20.877389 27820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:20.885567 27820 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:40313
I20260925 02:55:20.885596 27953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:40313 every 8 connection(s)
I20260925 02:55:20.886487 27820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:55:20.888859 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 27820
I20260925 02:55:20.889286 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:40297
--local_ip_for_outbound_sockets=127.12.222.195
--tserver_master_addrs=127.12.222.254:34743
--webserver_port=42575
--webserver_interface=127.12.222.195
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:55:20 dist-test-slave-252r krb5kdc[27043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304920, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:20.900215 27818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.887994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:41485 (local address 127.12.222.254:34743)
0925 02:55:20.888127 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:20.888132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:20.889040 (+ 908us) server_negotiation.cc:407] Connection header received
0925 02:55:20.890297 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:20.890303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:20.890369 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:20.890464 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:20.891902 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.892720 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.893820 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.894254 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.894372 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.896568 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:20.896575 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:20.896581 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:20.896617 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:20.898206 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:20.898651 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:20.898658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:20.898662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:20.898752 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:20.899057 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:20.899063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:20.899066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:20.899357 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:20.899458 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:20.899850 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 02:55:20.899999 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:55:20.900774 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:20.888455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:41485)
0925 02:55:20.888879 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:20.888938 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:20.890098 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:20.890661 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:20.890670 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:20.891132 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:20.891729 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.891749 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.892834 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.892837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.893692 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:20.893702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:20.894399 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:20.894401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:20.894520 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:20.895023 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:20.895037 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:20.896406 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:20.898324 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:20.898328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:20.898338 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:20.898555 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:20.898846 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:20.898848 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:20.898849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:20.898967 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:20.899447 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:20.899451 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:20.899633 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 02:55:20.899766 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 02:55:20.899973 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":135,"threads_started":1}
I20260925 02:55:20.901887 27954 heartbeater.cc:347] Connected to a master server at 127.12.222.254:34743
I20260925 02:55:20.902118 27954 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:20.902544 27954 heartbeater.cc:511] Master 127.12.222.254:34743 requested a full tablet report, sending...
I20260925 02:55:20.903486 27628 ts_manager.cc:194] Registered new tserver with Master: fed270065b044d3e99692d0663f3aab2 (127.12.222.194:40313)
I20260925 02:55:20.903982 27628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:41485
W20260925 02:55:20.992854 27957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:20.993077 27957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:20.993139 27957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:20.997169 27957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:20.997241 27957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:20.997366 27957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:55:21.002354 27957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:40297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=42575
--tserver_master_addrs=127.12.222.254:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:21.003223 27957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:21.004125 27957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:21.012746 27962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:21.012727 27963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:21.012730 27965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:21.012812 27957 server_base.cc:1034] running on GCE node
I20260925 02:55:21.013298 27957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:21.013901 27957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:21.015079 27957 hybrid_clock.cc:648] HybridClock initialized: now 1790304921015067 us; error 23 us; skew 500 ppm
Sep 25 02:55:21 dist-test-slave-252r krb5kdc[27043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304921, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:21.017541 27957 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:55:21.018302 27957 webserver.cc:460] Webserver started at http://127.12.222.195:42575/ using document root <none> and password file <none>
I20260925 02:55:21.018828 27957 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:21.018967 27957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:21.022369 27957 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:55:21.024364 27972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:21.025498 27957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:55:21.025624 27957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "3473197b94e8476f8143bb534ada9f09"
format_stamp: "Formatted at 2026-09-25 02:55:20 on dist-test-slave-252r"
server_key: "e51924c6a7472509cdc56e646669b032"
server_key_iv: "22382cda677f14071534882fe70a3506"
server_key_version: "encryptionkey@0"
I20260925 02:55:21.026124 27957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:21.045809 27957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:21.048341 27957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:21.048506 27957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:21.048992 27957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:21.049719 27957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:21.049788 27957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:21.049847 27957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:21.049902 27957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:21.058120 27957 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:40297
I20260925 02:55:21.058140 28085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:40297 every 8 connection(s)
I20260925 02:55:21.059002 27957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:55:21.065781 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 27957
Sep 25 02:55:21 dist-test-slave-252r krb5kdc[27043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304921, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:21.072057 27818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.060817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:45181 (local address 127.12.222.254:34743)
0925 02:55:21.061055 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:21.061059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:21.061938 (+ 879us) server_negotiation.cc:407] Connection header received
0925 02:55:21.063167 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:21.063170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:21.063223 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:21.063293 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:21.064338 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.064823 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.066000 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.066231 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.066316 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.068688 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:21.068692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:21.068694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:21.068719 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:21.070196 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:21.070651 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:21.070656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:21.070657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:21.070743 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:21.071051 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:21.071054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:21.071055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:21.071257 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:21.071356 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:21.071784 (+ 428us) server_negotiation.cc:299] Negotiation successful
0925 02:55:21.071899 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:55:21.072824 28088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.061265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:45181)
0925 02:55:21.061802 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:21.061841 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:21.062981 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:21.063472 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:21.063478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:21.063778 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:21.064172 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:21.064183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.064942 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.064945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.065865 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:21.065878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.066463 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.066469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.066673 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.067245 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:21.067259 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:21.068546 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:21.070296 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:21.070302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:21.070315 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:21.070553 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:21.070839 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:21.070841 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:21.070842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:21.070960 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:21.071369 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:21.071376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:21.071579 (+ 203us) client_negotiation.cc:769] Sending connection context
0925 02:55:21.071744 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 02:55:21.072004 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":137,"threads_started":1}
I20260925 02:55:21.073946 28086 heartbeater.cc:347] Connected to a master server at 127.12.222.254:34743
I20260925 02:55:21.074162 28086 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:21.074440 28090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.066464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55734 (local address 127.12.222.254:34743)
0925 02:55:21.066711 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:21.066716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:21.066846 (+ 130us) server_negotiation.cc:407] Connection header received
0925 02:55:21.066999 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:21.067002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:21.067047 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:21.067134 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:21.067808 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.068351 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.069470 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.069759 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.069832 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.071185 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:21.071188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:21.071189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:21.071208 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:21.072687 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:21.073231 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:21.073233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:21.073234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:21.073299 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:21.073557 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:21.073560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:21.073561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:21.073768 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:21.073999 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:21.074177 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 02:55:21.074293 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":96,"threads_started":1}
I20260925 02:55:21.074674 28086 heartbeater.cc:511] Master 127.12.222.254:34743 requested a full tablet report, sending...
I20260925 02:55:21.075667 27622 ts_manager.cc:194] Registered new tserver with Master: 3473197b94e8476f8143bb534ada9f09 (127.12.222.195:40297)
I20260925 02:55:21.076361 27622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:45181
I20260925 02:55:21.087481 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:55:21.091027 28090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.087979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55746 (local address 127.12.222.254:34743)
0925 02:55:21.088091 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:21.088094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:21.088303 (+ 209us) server_negotiation.cc:407] Connection header received
0925 02:55:21.088405 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:21.088411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:21.088484 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:21.088556 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:55:21.088758 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.089235 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.089950 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.090151 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.090217 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.090442 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:55:21.090718 (+ 276us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:55:21.090794 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:21.090843 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 02:55:21.090883 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:55:21.091536 27622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:55:21.094480 27622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:55:21.101064 27750 tablet_service.cc:1467] Processing CreateTablet for tablet d058255351e543e68aa968a0444cf24b (DEFAULT_TABLE table=test-table [id=24d34e8269ee4c58a576ddc233ee8b9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:21.101656 27750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d058255351e543e68aa968a0444cf24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:21.104215 27708 tablet_bootstrap.cc:492] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3: Bootstrap starting.
I20260925 02:55:21.104996 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.100140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:40297 (local address 127.0.0.1:52098)
0925 02:55:21.100216 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:21.100251 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:21.100312 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:21.101058 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:21.101061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:21.101077 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:21.101151 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:21.101157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.102336 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.102339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.103425 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:21.103436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.104553 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.104556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.104649 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.104704 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 02:55:21.104791 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 02:55:21.104886 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 02:55:21.105054 27708 tablet_bootstrap.cc:654] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:21.105496 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.100273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.12.222.195:40297)
0925 02:55:21.100581 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:21.100592 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:21.100611 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:55:21.100670 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:21.100675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:21.100866 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:21.101059 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:21.101250 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.102204 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.103548 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.104436 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.105185 (+ 749us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.105301 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:21.105340 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 02:55:21.105400 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":137,"threads_started":1}
I20260925 02:55:21.106307 28096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.100531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:40313 (local address 127.0.0.1:56454)
0925 02:55:21.100804 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:21.100818 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:21.100903 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:21.101943 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:21.101946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:21.101963 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:21.102051 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:21.102057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.103576 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.103579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.104614 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:21.104621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.105879 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.105883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.105983 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.106006 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:55:21.106096 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 02:55:21.106182 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":66,"threads_started":1}
I20260925 02:55:21.106422 28097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.100897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56454 (local address 127.12.222.194:40313)
0925 02:55:21.101521 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:21.101535 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:21.101553 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:55:21.101611 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:21.101615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:21.101799 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:21.101969 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:21.102157 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.103324 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.104727 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.105561 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.105692 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.105813 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:21.106174 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 02:55:21.106263 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":123,"threads_started":1}
I20260925 02:55:21.107340 27708 tablet_bootstrap.cc:492] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3: No bootstrap required, opened a new log
I20260925 02:55:21.107429 27708 ts_tablet_manager.cc:1397] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:55:21.107815 27708 raft_consensus.cc:348] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } } peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } }
I20260925 02:55:21.107936 27708 raft_consensus.cc:374] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:21.107972 27708 raft_consensus.cc:729] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bcc077b96d24bf8b37ced1d702717d3, State: Initialized, Role: FOLLOWER
I20260925 02:55:21.108083 27708 consensus_queue.cc:260] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } } peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } }
I20260925 02:55:21.108340 27708 ts_tablet_manager.cc:1428] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:55:21.108637 28020 tablet_service.cc:1467] Processing CreateTablet for tablet d058255351e543e68aa968a0444cf24b (DEFAULT_TABLE table=test-table [id=24d34e8269ee4c58a576ddc233ee8b9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:21.108956 27888 tablet_service.cc:1467] Processing CreateTablet for tablet d058255351e543e68aa968a0444cf24b (DEFAULT_TABLE table=test-table [id=24d34e8269ee4c58a576ddc233ee8b9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:21.109601 28020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d058255351e543e68aa968a0444cf24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:21.109930 27888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d058255351e543e68aa968a0444cf24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:21.116430 28099 tablet_bootstrap.cc:492] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09: Bootstrap starting.
I20260925 02:55:21.118762 28099 tablet_bootstrap.cc:654] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:21.119390 28099 log.cc:824] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:21.119899 28100 tablet_bootstrap.cc:492] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2: Bootstrap starting.
I20260925 02:55:21.121004 28099 tablet_bootstrap.cc:492] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09: No bootstrap required, opened a new log
I20260925 02:55:21.121145 28099 ts_tablet_manager.cc:1397] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:55:21.122167 28100 tablet_bootstrap.cc:654] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:21.122812 28100 log.cc:824] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:21.123842 28099 raft_consensus.cc:348] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } } peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } }
I20260925 02:55:21.124028 28099 raft_consensus.cc:374] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:21.124065 28099 raft_consensus.cc:729] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3473197b94e8476f8143bb534ada9f09, State: Initialized, Role: FOLLOWER
I20260925 02:55:21.124447 28100 tablet_bootstrap.cc:492] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2: No bootstrap required, opened a new log
I20260925 02:55:21.124469 28099 consensus_queue.cc:260] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } } peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } }
I20260925 02:55:21.124608 28100 ts_tablet_manager.cc:1397] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:55:21.125098 28086 heartbeater.cc:503] Master 127.12.222.254:34743 was elected leader, sending a full tablet report...
I20260925 02:55:21.125295 28099 ts_tablet_manager.cc:1428] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 02:55:21.127841 28100 raft_consensus.cc:348] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } } peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } }
I20260925 02:55:21.128090 28100 raft_consensus.cc:374] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:21.128145 28100 raft_consensus.cc:729] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed270065b044d3e99692d0663f3aab2, State: Initialized, Role: FOLLOWER
I20260925 02:55:21.128538 28100 consensus_queue.cc:260] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } } peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } }
I20260925 02:55:21.129136 27954 heartbeater.cc:503] Master 127.12.222.254:34743 was elected leader, sending a full tablet report...
I20260925 02:55:21.129410 28100 ts_tablet_manager.cc:1428] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260925 02:55:21.137267 27955 tablet.cc:2369] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:55:21.209434 27817 tablet.cc:2369] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:55:21.287134 28103 raft_consensus.cc:482] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:55:21.287314 28103 raft_consensus.cc:504] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } } peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } }
I20260925 02:55:21.288609 28103 leader_election.cc:288] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fed270065b044d3e99692d0663f3aab2 (127.12.222.194:40313), 5bcc077b96d24bf8b37ced1d702717d3 (127.12.222.193:40341)
I20260925 02:55:21.293695 28097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.289240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:34237 (local address 127.12.222.194:40313)
0925 02:55:21.289399 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:21.289405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:21.289420 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:55:21.289480 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:21.289501 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:21.289556 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:21.289640 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:21.289962 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.290727 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.292379 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.292988 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.293116 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.293170 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:21.293489 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 02:55:21.293563 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:55:21.293699 28106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.289073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:59075 (local address 127.12.222.193:40341)
0925 02:55:21.289364 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:21.289369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:21.289419 (+ 50us) server_negotiation.cc:407] Connection header received
0925 02:55:21.289554 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:21.289557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:21.289624 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:21.289728 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:21.289970 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.290727 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.292379 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.293027 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.293132 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.293291 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:21.293489 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 02:55:21.293564 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":149,"threads_started":1}
I20260925 02:55:21.293736 28088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.288792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:40313 (local address 127.12.222.195:34237)
0925 02:55:21.288984 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:21.289000 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:21.289085 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:21.289654 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:21.289656 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:21.289669 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:21.289819 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:21.289829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.290837 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.290841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.292251 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:21.292259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.293132 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.293136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.293330 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.293372 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 02:55:21.293452 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 02:55:21.293577 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":168}
I20260925 02:55:21.293744 28105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.288856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:40341 (local address 127.12.222.195:59075)
0925 02:55:21.289333 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:21.289348 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:21.289453 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:21.289743 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:21.289745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:21.289756 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:21.289852 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:21.289861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.290837 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.290840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.292247 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:21.292259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.293132 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.293136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:21.293325 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.293371 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:55:21.293452 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 02:55:21.293577 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":158,"thread_start_us":110,"threads_started":1}
I20260925 02:55:21.294427 27770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d058255351e543e68aa968a0444cf24b" candidate_uuid: "3473197b94e8476f8143bb534ada9f09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bcc077b96d24bf8b37ced1d702717d3" is_pre_election: true
I20260925 02:55:21.294463 27908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d058255351e543e68aa968a0444cf24b" candidate_uuid: "3473197b94e8476f8143bb534ada9f09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed270065b044d3e99692d0663f3aab2" is_pre_election: true
I20260925 02:55:21.294706 27770 raft_consensus.cc:2454] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3473197b94e8476f8143bb534ada9f09 in term 0.
I20260925 02:55:21.294754 27908 raft_consensus.cc:2454] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3473197b94e8476f8143bb534ada9f09 in term 0.
I20260925 02:55:21.295245 27975 leader_election.cc:302] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3473197b94e8476f8143bb534ada9f09, 5bcc077b96d24bf8b37ced1d702717d3; no voters:
I20260925 02:55:21.295531 28103 raft_consensus.cc:2781] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:55:21.295619 28103 raft_consensus.cc:482] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:55:21.295651 28103 raft_consensus.cc:3037] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:21.296627 28103 raft_consensus.cc:504] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } } peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } }
I20260925 02:55:21.296955 28103 leader_election.cc:288] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [CANDIDATE]: Term 1 election: Requested vote from peers fed270065b044d3e99692d0663f3aab2 (127.12.222.194:40313), 5bcc077b96d24bf8b37ced1d702717d3 (127.12.222.193:40341)
I20260925 02:55:21.297222 27908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d058255351e543e68aa968a0444cf24b" candidate_uuid: "3473197b94e8476f8143bb534ada9f09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed270065b044d3e99692d0663f3aab2"
I20260925 02:55:21.297284 27770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d058255351e543e68aa968a0444cf24b" candidate_uuid: "3473197b94e8476f8143bb534ada9f09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bcc077b96d24bf8b37ced1d702717d3"
I20260925 02:55:21.297367 27908 raft_consensus.cc:3037] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:21.297421 27770 raft_consensus.cc:3037] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:21.298352 27908 raft_consensus.cc:2454] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3473197b94e8476f8143bb534ada9f09 in term 1.
I20260925 02:55:21.298420 27770 raft_consensus.cc:2454] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3473197b94e8476f8143bb534ada9f09 in term 1.
I20260925 02:55:21.298648 27973 leader_election.cc:302] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3473197b94e8476f8143bb534ada9f09, fed270065b044d3e99692d0663f3aab2; no voters:
I20260925 02:55:21.298812 28103 raft_consensus.cc:2781] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:21.299050 28103 raft_consensus.cc:686] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [term 1 LEADER]: Becoming Leader. State: Replica: 3473197b94e8476f8143bb534ada9f09, State: Running, Role: LEADER
I20260925 02:55:21.299386 28103 consensus_queue.cc:237] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } } peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } }
I20260925 02:55:21.302307 27622 catalog_manager.cc:5495] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3473197b94e8476f8143bb534ada9f09 (127.12.222.195). New cstate: current_term: 1 leader_uuid: "3473197b94e8476f8143bb534ada9f09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } health_report { overall_health: UNKNOWN } } }
W20260925 02:55:21.309911 28087 tablet.cc:2369] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:55:21.341948 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.339005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52104 (local address 127.12.222.195:40297)
0925 02:55:21.339136 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:21.339139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:21.339151 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:55:21.339219 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:21.339221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:21.339270 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:21.339354 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:55:21.339553 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.340022 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.340769 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:21.340974 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:21.341045 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:21.341287 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:55:21.341678 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:55:21.341758 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:21.341811 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 02:55:21.341854 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:55:21.347273 27770 raft_consensus.cc:1261] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Refusing update from remote peer 3473197b94e8476f8143bb534ada9f09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:21.347277 27908 raft_consensus.cc:1261] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2 [term 1 FOLLOWER]: Refusing update from remote peer 3473197b94e8476f8143bb534ada9f09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:21.347916 28103 consensus_queue.cc:1035] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:21.348073 28107 consensus_queue.cc:1035] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:21.353889 28111 mvcc.cc:205] Tried to move back new op lower bound from 7333088957831798784 to 7333088957644730368. Current Snapshot: MvccSnapshot[applied={T|T < 7333088957831798784}]
I20260925 02:55:21.362443 27628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:21.362584 27628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:21.364559 27628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64: Sending DeleteTablet for 3 replicas of tablet d058255351e543e68aa968a0444cf24b
I20260925 02:55:21.365196 27888 tablet_service.cc:1514] Processing DeleteTablet for tablet d058255351e543e68aa968a0444cf24b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:21 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:56454
I20260925 02:55:21.365198 28020 tablet_service.cc:1514] Processing DeleteTablet for tablet d058255351e543e68aa968a0444cf24b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:21 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:52098
I20260925 02:55:21.365567 27750 tablet_service.cc:1514] Processing DeleteTablet for tablet d058255351e543e68aa968a0444cf24b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:21 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:42034
I20260925 02:55:21.365597 28120 tablet_replica.cc:330] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09: stopping tablet replica
I20260925 02:55:21.365778 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 27666
I20260925 02:55:21.365867 28120 raft_consensus.cc:2229] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:55:21.365902 28122 tablet_replica.cc:330] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3: stopping tablet replica
I20260925 02:55:21.366014 28122 raft_consensus.cc:2229] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:21.366142 28122 raft_consensus.cc:2258] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:21.366170 28120 raft_consensus.cc:2258] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:21.366520 28121 tablet_replica.cc:330] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2: stopping tablet replica
I20260925 02:55:21.366827 28121 raft_consensus.cc:2229] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:21.367141 28121 raft_consensus.cc:2258] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:21.367976 28120 ts_tablet_manager.cc:1905] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:21.368465 28121 ts_tablet_manager.cc:1905] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:21.370939 28121 ts_tablet_manager.cc:1918] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:21.371080 28121 log.cc:1196] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/d058255351e543e68aa968a0444cf24b
I20260925 02:55:21.371176 28120 ts_tablet_manager.cc:1918] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:21.371332 28120 log.cc:1196] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/d058255351e543e68aa968a0444cf24b
I20260925 02:55:21.371465 28121 ts_tablet_manager.cc:1939] T d058255351e543e68aa968a0444cf24b P fed270065b044d3e99692d0663f3aab2: Deleting consensus metadata
I20260925 02:55:21.371768 28120 ts_tablet_manager.cc:1939] T d058255351e543e68aa968a0444cf24b P 3473197b94e8476f8143bb534ada9f09: Deleting consensus metadata
I20260925 02:55:21.372635 27612 catalog_manager.cc:4841] TS fed270065b044d3e99692d0663f3aab2 (127.12.222.194:40313): tablet d058255351e543e68aa968a0444cf24b (table test-table [id=24d34e8269ee4c58a576ddc233ee8b9a]) successfully deleted
I20260925 02:55:21.372825 27615 catalog_manager.cc:4841] TS 3473197b94e8476f8143bb534ada9f09 (127.12.222.195:40297): tablet d058255351e543e68aa968a0444cf24b (table test-table [id=24d34e8269ee4c58a576ddc233ee8b9a]) successfully deleted
W20260925 02:55:21.375839 27614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.222.193:40341 (error 108)
I20260925 02:55:21.375912 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 27820
I20260925 02:55:21.381494 28096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.381193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:40341 (local address 127.0.0.1:42044)
0925 02:55:21.381336 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:21.381416 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.193:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 02:55:21.381655 27614 catalog_manager.cc:4584] TS 5bcc077b96d24bf8b37ced1d702717d3 (127.12.222.193:40341): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d058255351e543e68aa968a0444cf24b: Network error: Client connection negotiation failed: client connection to 127.12.222.193:40341: connect: Connection refused (error 111)
I20260925 02:55:21.382462 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 27957
I20260925 02:55:21.387931 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 27595
I20260925 02:55:21.393853 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:34743
--webserver_interface=127.12.222.254
--webserver_port=37109
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:34743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:21.492902 28124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:21.493094 28124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:21.493129 28124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:21.496682 28124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:21.496721 28124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:21.496738 28124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:21.496752 28124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:21.496766 28124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:21.501309 28124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:34743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:34743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=37109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:21.502055 28124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:21.502934 28124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:21.510403 28130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:21.510413 28129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:21.510535 28124 server_base.cc:1034] running on GCE node
W20260925 02:55:21.510433 28132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:21.511158 28124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:21.512079 28124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:21.513257 28124 hybrid_clock.cc:648] HybridClock initialized: now 1790304921513231 us; error 49 us; skew 500 ppm
Sep 25 02:55:21 dist-test-slave-252r krb5kdc[27043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304921, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:21.515672 28124 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:55:21.516515 28124 webserver.cc:460] Webserver started at http://127.12.222.254:37109/ using document root <none> and password file <none>
I20260925 02:55:21.516952 28124 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:21.517006 28124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:21.520421 28124 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 02:55:21.522305 28139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:21.523535 28124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:55:21.523636 28124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "00aaad83295f4139b7f27b616d20de64"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-252r"
server_key: "7b8015f0f1e449df50c207b4ed7b0f48"
server_key_iv: "ba18287f4dce18c78d499b5154978176"
server_key_version: "encryptionkey@0"
I20260925 02:55:21.524173 28124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:21.533834 28124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:21.536156 28124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:21.536366 28124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:21.542609 28124 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:34743
I20260925 02:55:21.542860 28191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:34743 every 8 connection(s)
I20260925 02:55:21.543537 28124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:21.547148 28192 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:55:21.548650 28192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64: Bootstrap starting.
I20260925 02:55:21.549108 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 28124
I20260925 02:55:21.549512 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:40341
--local_ip_for_outbound_sockets=127.12.222.193
--tserver_master_addrs=127.12.222.254:34743
--webserver_port=32829
--webserver_interface=127.12.222.193
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:55:21.564548 28192 log.cc:824] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:21.574347 28192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:55:21.574702 28192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64: Bootstrap complete.
I20260925 02:55:21.578161 28192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00aaad83295f4139b7f27b616d20de64" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 34743 } }
I20260925 02:55:21.578532 28192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00aaad83295f4139b7f27b616d20de64, State: Initialized, Role: FOLLOWER
I20260925 02:55:21.578992 28192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00aaad83295f4139b7f27b616d20de64" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 34743 } }
I20260925 02:55:21.579123 28192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:21.579202 28192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:21.579315 28192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:55:21.580996 28192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00aaad83295f4139b7f27b616d20de64" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 34743 } }
I20260925 02:55:21.581295 28192 leader_election.cc:302] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00aaad83295f4139b7f27b616d20de64; no voters:
I20260925 02:55:21.581574 28192 leader_election.cc:288] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 02:55:21.581702 28196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 02:55:21.581925 28196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [term 3 LEADER]: Becoming Leader. State: Replica: 00aaad83295f4139b7f27b616d20de64, State: Running, Role: LEADER
I20260925 02:55:21.582199 28196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00aaad83295f4139b7f27b616d20de64" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 34743 } }
I20260925 02:55:21.582618 28192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:21.583107 28198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00aaad83295f4139b7f27b616d20de64. Latest consensus state: current_term: 3 leader_uuid: "00aaad83295f4139b7f27b616d20de64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00aaad83295f4139b7f27b616d20de64" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 34743 } } }
I20260925 02:55:21.583242 28198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:21.583406 28197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "00aaad83295f4139b7f27b616d20de64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00aaad83295f4139b7f27b616d20de64" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 34743 } } }
I20260925 02:55:21.583516 28197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00aaad83295f4139b7f27b616d20de64 [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:21.584703 28201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:21.588168 28201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:21.588856 28201 catalog_manager.cc:1259] Loaded cluster ID: 7ba504fd0cb84d4aa63a47ff0bb6794f
I20260925 02:55:21.588919 28201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 02:55:21.589677 28201 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f90a14379ad google::LogMessage::Fail() at ??:0
@ 0x7f90a143a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f90a14374de google::LogMessage::Flush() at ??:0
@ 0x7f90a1438a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f90abd76fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f90abd77d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f90abd74579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f90abdb8640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f90a36893d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f90a3689c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f90a368b3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f90a367a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f90a4cbe6db start_thread at ??:0
@ 0x7f909ff3561f clone at ??:0
W20260925 02:55:21.679109 28194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:21.679388 28194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:21.679453 28194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:21.685138 28194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:21.685206 28194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:21.685286 28194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:55:21.692677 28194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:40341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=32829
--tserver_master_addrs=127.12.222.254:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:21.693843 28194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:21.695036 28194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:21.705817 28233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:21.705825 28234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:21.705825 28236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:21.706306 28194 server_base.cc:1034] running on GCE node
I20260925 02:55:21.706732 28194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:21.707459 28194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:21.708654 28194 hybrid_clock.cc:648] HybridClock initialized: now 1790304921708616 us; error 44 us; skew 500 ppm
Sep 25 02:55:21 dist-test-slave-252r krb5kdc[27043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304921, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:21.711730 28194 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:55:21.712738 28194 webserver.cc:460] Webserver started at http://127.12.222.193:32829/ using document root <none> and password file <none>
I20260925 02:55:21.713371 28194 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:21.713500 28194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:21.718386 28194 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:55:21.720875 28243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:21.722302 28194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:55:21.722411 28194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "5bcc077b96d24bf8b37ced1d702717d3"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-252r"
server_key: "47c7c84771d59213bdf021b1b6f31f02"
server_key_iv: "7544f6af651365d9eea746bdaebeb4e6"
server_key_version: "encryptionkey@0"
I20260925 02:55:21.723078 28194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:21.747529 28194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:21.750308 28194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:21.750519 28194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:21.751073 28194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:21.752080 28250 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 02:55:21.756768 28194 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:55:21.756839 28194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260925 02:55:21.756920 28194 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:55:21.758350 28194 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:55:21.758412 28194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 02:55:21.758723 28250 tablet_bootstrap.cc:492] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3: Bootstrap starting.
I20260925 02:55:21.768615 28194 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:40341
I20260925 02:55:21.768646 28357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:40341 every 8 connection(s)
I20260925 02:55:21.769464 28194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:55:21.776173 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 28194
I20260925 02:55:21.776674 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:40313
--local_ip_for_outbound_sockets=127.12.222.194
--tserver_master_addrs=127.12.222.254:34743
--webserver_port=34483
--webserver_interface=127.12.222.194
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:55:21.778746 28250 log.cc:824] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:21.787096 28250 tablet_bootstrap.cc:492] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:55:21.787485 28250 tablet_bootstrap.cc:492] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3: Bootstrap complete.
I20260925 02:55:21.788859 28250 ts_tablet_manager.cc:1397] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3: Time spent bootstrapping tablet: real 0.030s user 0.019s sys 0.010s
I20260925 02:55:21.791814 28250 raft_consensus.cc:348] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } } peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } }
I20260925 02:55:21.792330 28250 raft_consensus.cc:729] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bcc077b96d24bf8b37ced1d702717d3, State: Initialized, Role: FOLLOWER
I20260925 02:55:21.792833 28250 consensus_queue.cc:260] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } } peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } }
I20260925 02:55:21.793812 28250 ts_tablet_manager.cc:1428] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260925 02:55:21.828068 28360 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:55:21.771075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:40101)
0925 02:55:21.771516 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:21.771562 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:21.772344 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:21.827135 (+ 54791us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":157,"threads_started":1}
W20260925 02:55:21.828557 28358 heartbeater.cc:644] Failed to heartbeat to 127.12.222.254:34743 (0 consecutive failures): Network error: Failed to ping master at 127.12.222.254:34743: Client connection negotiation failed: client connection to 127.12.222.254:34743: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:55:21.829236 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.828998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:39063)
0925 02:55:21.829116 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:21.829180 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 02:55:21.829876 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.829684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:44781)
0925 02:55:21.829793 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:21.829833 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 02:55:21.830153 28358 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:55:21.878758 28361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:21.878942 28361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:21.878974 28361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:21.882689 28361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:21.882732 28361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:21.882823 28361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:55:21.887377 28361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:40313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=34483
--tserver_master_addrs=127.12.222.254:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:21.888182 28361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:21.889041 28361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:21.897420 28368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:21.897439 28371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:21.897439 28369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:21.897501 28361 server_base.cc:1034] running on GCE node
I20260925 02:55:21.898041 28361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:21.898561 28361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:21.899729 28361 hybrid_clock.cc:648] HybridClock initialized: now 1790304921899704 us; error 35 us; skew 500 ppm
Sep 25 02:55:21 dist-test-slave-252r krb5kdc[27043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304921, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:21.902281 28361 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:55:21.903074 28361 webserver.cc:460] Webserver started at http://127.12.222.194:34483/ using document root <none> and password file <none>
I20260925 02:55:21.903563 28361 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:21.903644 28361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:21.907002 28361 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:55:21.908787 28378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:21.909763 28361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:55:21.909883 28361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "fed270065b044d3e99692d0663f3aab2"
format_stamp: "Formatted at 2026-09-25 02:55:19 on dist-test-slave-252r"
server_key: "8be58501d7b0276e50968a992ca140fa"
server_key_iv: "5c85abbff519cee9da3ad7c644563b0c"
server_key_version: "encryptionkey@0"
I20260925 02:55:21.910324 28361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:21.923202 28361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:21.925987 28361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:21.926177 28361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:21.926673 28361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:21.927464 28361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:21.927531 28361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:21.927599 28361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:21.927644 28361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:21.935899 28361 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:40313
I20260925 02:55:21.935930 28491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:40313 every 8 connection(s)
I20260925 02:55:21.936774 28361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:55:21.939915 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.938496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:45985)
0925 02:55:21.938855 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:21.938951 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":96,"threads_started":1}
W20260925 02:55:21.940328 28492 heartbeater.cc:644] Failed to heartbeat to 127.12.222.254:34743 (0 consecutive failures): Network error: Failed to ping master at 127.12.222.254:34743: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
I20260925 02:55:21.940862 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.940633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:43481)
0925 02:55:21.940732 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:21.940776 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 02:55:21.941519 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:21.941314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:60407)
0925 02:55:21.941424 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:21.941465 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 02:55:21.941694 28492 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:55:21.942842 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 28361
I20260925 02:55:21.943208 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:40297
--local_ip_for_outbound_sockets=127.12.222.195
--tserver_master_addrs=127.12.222.254:34743
--webserver_port=42575
--webserver_interface=127.12.222.195
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:22.044785 28495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:22.044979 28495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:22.045014 28495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:22.048792 28495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:22.048833 28495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:22.048882 28495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:55:22.053333 28495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:40297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=42575
--tserver_master_addrs=127.12.222.254:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:22.054101 28495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:22.054924 28495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:22.063627 28501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:22.063665 28500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:22.063659 28503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:22.063709 28495 server_base.cc:1034] running on GCE node
I20260925 02:55:22.064137 28495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:22.064728 28495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:22.065936 28495 hybrid_clock.cc:648] HybridClock initialized: now 1790304922065910 us; error 39 us; skew 500 ppm
Sep 25 02:55:22 dist-test-slave-252r krb5kdc[27043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304922, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:22.068497 28495 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:55:22.069217 28495 webserver.cc:460] Webserver started at http://127.12.222.195:42575/ using document root <none> and password file <none>
I20260925 02:55:22.069751 28495 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:22.069840 28495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:22.073053 28495 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:55:22.074756 28510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:22.075784 28495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 02:55:22.075904 28495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "3473197b94e8476f8143bb534ada9f09"
format_stamp: "Formatted at 2026-09-25 02:55:20 on dist-test-slave-252r"
server_key: "e51924c6a7472509cdc56e646669b032"
server_key_iv: "22382cda677f14071534882fe70a3506"
server_key_version: "encryptionkey@0"
I20260925 02:55:22.076455 28495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:22.084563 28495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:22.087673 28495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:22.087857 28495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:22.088419 28495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:22.089123 28495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:22.089191 28495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:22.089254 28495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:22.089303 28495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:22.097076 28495 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:40297
I20260925 02:55:22.097143 28623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:40297 every 8 connection(s)
I20260925 02:55:22.097957 28495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:55:22.098619 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 28495
I20260925 02:55:22.101471 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:22.099848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:36047)
0925 02:55:22.100213 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:22.100407 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":99,"threads_started":1}
W20260925 02:55:22.101863 28624 heartbeater.cc:644] Failed to heartbeat to 127.12.222.254:34743 (0 consecutive failures): Network error: Failed to ping master at 127.12.222.254:34743: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
I20260925 02:55:22.102391 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:22.102206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:48943)
0925 02:55:22.102294 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:22.102347 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 02:55:22.102900 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:22.102737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:35971)
0925 02:55:22.102808 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:22.102848 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 02:55:22.103070 28624 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:55:22.831015 28628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:22.830632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:60001)
0925 02:55:22.830855 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:22.830910 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":67,"threads_started":1}
I20260925 02:55:22.942523 28629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:22.942175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:36473)
0925 02:55:22.942390 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:22.942438 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":62,"threads_started":1}
I20260925 02:55:23.103919 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:23.103555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:40049)
0925 02:55:23.103782 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:23.103827 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
I20260925 02:55:23.565750 28631 raft_consensus.cc:482] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:55:23.565965 28631 raft_consensus.cc:504] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } } peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } }
I20260925 02:55:23.567003 28631 leader_election.cc:288] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3473197b94e8476f8143bb534ada9f09 (127.12.222.195:40297), fed270065b044d3e99692d0663f3aab2 (127.12.222.194:40313)
Sep 25 02:55:23 dist-test-slave-252r krb5kdc[27043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304921, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.195@KRBTEST.COM
Sep 25 02:55:23 dist-test-slave-252r krb5kdc[27043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304921, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.194@KRBTEST.COM
I20260925 02:55:23.583500 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:23.567188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:40297 (local address 127.12.222.193:39513)
0925 02:55:23.567606 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:23.567617 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:23.567681 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:23.569604 (+ 1923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:23.569611 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:23.570204 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:23.571028 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:23.571049 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:23.572838 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:23.572841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:23.574340 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:23.574352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:23.575181 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:23.575184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:23.575490 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:23.576192 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:23.576215 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:23.578163 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:23.580562 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:23.580567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:23.580580 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:23.581063 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:23.581453 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:23.581456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:23.581458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:23.581660 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:23.582400 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:23.582402 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:23.582731 (+ 329us) client_negotiation.cc:769] Sending connection context
0925 02:55:23.582963 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 02:55:23.583291 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":538,"spinlock_wait_cycles":73600,"thread_start_us":42,"threads_started":1}
I20260925 02:55:23.583552 28635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:23.567367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:57457 (local address 127.12.222.194:40313)
0925 02:55:23.567765 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:23.567780 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:23.567821 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:55:23.568719 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:23.568727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:23.569166 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:23.569509 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:23.571203 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:23.572684 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:23.574453 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:23.575068 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:23.575216 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:23.578661 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:23.578669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:23.578682 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:23.578715 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:23.580445 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:23.581179 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:23.581185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:23.581190 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:23.581344 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:23.581760 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:23.581763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:23.581764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:23.582253 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:23.582388 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:23.583106 (+ 718us) server_negotiation.cc:299] Negotiation successful
0925 02:55:23.583373 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":106,"threads_started":1}
I20260925 02:55:23.583555 28634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:23.567367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:39513 (local address 127.12.222.195:40297)
0925 02:55:23.567765 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:23.567783 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:23.567821 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:55:23.568719 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:23.568727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:23.569165 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:23.569509 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:23.571203 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:23.572684 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:23.574453 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:23.575068 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:23.575216 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:23.578398 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:23.578403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:23.578419 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:23.578443 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:23.580434 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:23.581179 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:23.581185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:23.581190 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:23.581344 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:23.581760 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:23.581763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:23.581764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:23.582253 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:23.582388 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:23.583106 (+ 718us) server_negotiation.cc:299] Negotiation successful
0925 02:55:23.583373 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":92,"threads_started":1}
I20260925 02:55:23.583500 28632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:23.567188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:40313 (local address 127.12.222.193:57457)
0925 02:55:23.567529 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:23.567543 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:23.567646 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:23.569604 (+ 1958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:23.569611 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:23.570204 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:23.571028 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:23.571049 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:23.572838 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:23.572841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:23.574340 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:23.574352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:23.575181 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:23.575184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:23.575491 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:23.576190 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:23.576210 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:23.578476 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:23.580562 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:23.580567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:23.580580 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:23.581063 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:23.581453 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:23.581456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:23.581458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:23.581660 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:23.582394 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:23.582400 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:23.582731 (+ 331us) client_negotiation.cc:769] Sending connection context
0925 02:55:23.582963 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 02:55:23.583342 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":466,"spinlock_wait_cycles":20864,"thread_start_us":73,"threads_started":1}
W20260925 02:55:23.586263 28244 leader_election.cc:341] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fed270065b044d3e99692d0663f3aab2 (127.12.222.194:40313): Not found: Tablet not found: d058255351e543e68aa968a0444cf24b
W20260925 02:55:23.586494 28247 leader_election.cc:341] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3473197b94e8476f8143bb534ada9f09 (127.12.222.195:40297): Not found: Tablet not found: d058255351e543e68aa968a0444cf24b
I20260925 02:55:23.586575 28247 leader_election.cc:302] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5bcc077b96d24bf8b37ced1d702717d3; no voters: 3473197b94e8476f8143bb534ada9f09, fed270065b044d3e99692d0663f3aab2
I20260925 02:55:23.586855 28631 raft_consensus.cc:2726] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:55:23.831969 28632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:23.831652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:55351)
0925 02:55:23.831813 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:23.831874 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 02:55:23.943747 28636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:23.943381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:59013)
0925 02:55:23.943603 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:23.943652 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":64,"threads_started":1}
I20260925 02:55:24.104988 28637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.104659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:60355)
0925 02:55:24.104857 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:24.104905 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":59,"threads_started":1}
I20260925 02:55:24.833004 28638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.832620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:39993)
0925 02:55:24.832849 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:24.832910 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
I20260925 02:55:24.944885 28639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:24.944555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:40145)
0925 02:55:24.944763 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:24.944810 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":63,"threads_started":1}
I20260925 02:55:25.106009 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:25.105607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:45407)
0925 02:55:25.105840 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:25.105904 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":65,"threads_started":1}
I20260925 02:55:25.337720 28641 raft_consensus.cc:482] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:55:25.337837 28641 raft_consensus.cc:504] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } } peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } }
I20260925 02:55:25.338246 28641 leader_election.cc:288] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3473197b94e8476f8143bb534ada9f09 (127.12.222.195:40297), fed270065b044d3e99692d0663f3aab2 (127.12.222.194:40313)
W20260925 02:55:25.338936 28247 leader_election.cc:341] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3473197b94e8476f8143bb534ada9f09 (127.12.222.195:40297): Not found: Tablet not found: d058255351e543e68aa968a0444cf24b
W20260925 02:55:25.339066 28244 leader_election.cc:341] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fed270065b044d3e99692d0663f3aab2 (127.12.222.194:40313): Not found: Tablet not found: d058255351e543e68aa968a0444cf24b
I20260925 02:55:25.339161 28244 leader_election.cc:302] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5bcc077b96d24bf8b37ced1d702717d3; no voters: 3473197b94e8476f8143bb534ada9f09, fed270065b044d3e99692d0663f3aab2
I20260925 02:55:25.339298 28641 raft_consensus.cc:2726] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:55:25.834172 28642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:25.833787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:52765)
0925 02:55:25.834010 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:25.834079 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260925 02:55:25.945894 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:25.945498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:50287)
0925 02:55:25.945733 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:25.945790 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
I20260925 02:55:26.107040 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:26.106713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:49021)
0925 02:55:26.106924 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:26.106970 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":69,"threads_started":1}
I20260925 02:55:26.835491 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:26.835071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:55915)
0925 02:55:26.835347 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:26.835399 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":71,"threads_started":1}
I20260925 02:55:26.946982 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:26.946603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:41093)
0925 02:55:26.946835 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:26.946884 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":82,"threads_started":1}
I20260925 02:55:27.108106 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:27.107703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:39165)
0925 02:55:27.107990 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:27.108032 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":65,"threads_started":1}
I20260925 02:55:27.836719 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:27.836312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:53407)
0925 02:55:27.836554 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:27.836607 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260925 02:55:27.855839 28649 raft_consensus.cc:482] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:55:27.855953 28649 raft_consensus.cc:504] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } } peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } }
I20260925 02:55:27.856387 28649 leader_election.cc:288] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3473197b94e8476f8143bb534ada9f09 (127.12.222.195:40297), fed270065b044d3e99692d0663f3aab2 (127.12.222.194:40313)
W20260925 02:55:27.857127 28244 leader_election.cc:341] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fed270065b044d3e99692d0663f3aab2 (127.12.222.194:40313): Not found: Tablet not found: d058255351e543e68aa968a0444cf24b
W20260925 02:55:27.857208 28247 leader_election.cc:341] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3473197b94e8476f8143bb534ada9f09 (127.12.222.195:40297): Not found: Tablet not found: d058255351e543e68aa968a0444cf24b
I20260925 02:55:27.857247 28244 leader_election.cc:302] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5bcc077b96d24bf8b37ced1d702717d3; no voters: 3473197b94e8476f8143bb534ada9f09, fed270065b044d3e99692d0663f3aab2
I20260925 02:55:27.857406 28649 raft_consensus.cc:2726] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:55:27.948063 28650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:27.947689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:45279)
0925 02:55:27.947940 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:27.947987 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":84,"threads_started":1}
I20260925 02:55:28.109012 28651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.108680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:52623)
0925 02:55:28.108889 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:28.108937 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":64,"threads_started":1}
I20260925 02:55:28.837826 28652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.837431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:57141)
0925 02:55:28.837681 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:28.837732 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":69,"threads_started":1}
I20260925 02:55:28.949186 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:28.948793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:46461)
0925 02:55:28.949067 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:28.949113 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":70,"threads_started":1}
I20260925 02:55:29.110085 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:29.109614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:47883)
0925 02:55:29.109962 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:29.110011 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":58,"threads_started":1}
I20260925 02:55:29.839001 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:29.838619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:36641)
0925 02:55:29.838842 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:29.838895 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":67,"threads_started":1}
I20260925 02:55:29.950320 28656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:29.949943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:57257)
0925 02:55:29.950194 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:29.950242 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":110,"threads_started":1}
I20260925 02:55:30.111127 28657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:30.110748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:55139)
0925 02:55:30.110989 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:30.111052 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":64,"threads_started":1}
I20260925 02:55:30.840282 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:30.839864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:47201)
0925 02:55:30.840083 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:30.840130 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
I20260925 02:55:30.951442 28659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:30.951058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:42875)
0925 02:55:30.951300 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:30.951350 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":88,"threads_started":1}
I20260925 02:55:31.112202 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.111859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:55005)
0925 02:55:31.112081 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:31.112128 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
I20260925 02:55:31.484412 28661 raft_consensus.cc:482] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:55:31.484581 28661 raft_consensus.cc:504] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } } peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } }
I20260925 02:55:31.485091 28661 leader_election.cc:288] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3473197b94e8476f8143bb534ada9f09 (127.12.222.195:40297), fed270065b044d3e99692d0663f3aab2 (127.12.222.194:40313)
W20260925 02:55:31.485797 28247 leader_election.cc:341] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3473197b94e8476f8143bb534ada9f09 (127.12.222.195:40297): Not found: Tablet not found: d058255351e543e68aa968a0444cf24b
W20260925 02:55:31.485917 28244 leader_election.cc:341] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fed270065b044d3e99692d0663f3aab2 (127.12.222.194:40313): Not found: Tablet not found: d058255351e543e68aa968a0444cf24b
I20260925 02:55:31.485998 28244 leader_election.cc:302] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5bcc077b96d24bf8b37ced1d702717d3; no voters: 3473197b94e8476f8143bb534ada9f09, fed270065b044d3e99692d0663f3aab2
I20260925 02:55:31.486132 28661 raft_consensus.cc:2726] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:55:31.841497 28662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.841081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:54953)
0925 02:55:31.841334 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:31.841401 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":98,"threads_started":1}
I20260925 02:55:31.952711 28663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:31.952333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:43813)
0925 02:55:31.952577 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:31.952627 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":90,"threads_started":1}
I20260925 02:55:32.113190 28664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:32.112855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:34249)
0925 02:55:32.113069 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:32.113116 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":73,"threads_started":1}
I20260925 02:55:32.842543 28665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:32.842159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:37099)
0925 02:55:32.842403 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:32.842450 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
I20260925 02:55:32.953770 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:32.953425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:56809)
0925 02:55:32.953648 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:32.953695 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":77,"threads_started":1}
I20260925 02:55:33.114233 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:33.113873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:50789)
0925 02:55:33.114108 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:33.114155 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":109,"threads_started":1}
I20260925 02:55:33.843704 28668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:33.843233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:36979)
0925 02:55:33.843545 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:33.843599 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":86,"threads_started":1}
I20260925 02:55:33.954828 28669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:33.954439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:44941)
0925 02:55:33.954706 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:33.954752 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
I20260925 02:55:34.115149 28670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:34.114824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:40087)
0925 02:55:34.115030 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:34.115076 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":70,"threads_started":1}
I20260925 02:55:34.844794 28671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:34.844435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:49327)
0925 02:55:34.844655 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:34.844703 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260925 02:55:34.955821 28672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:34.955474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:55595)
0925 02:55:34.955692 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:34.955742 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":63,"threads_started":1}
I20260925 02:55:35.071609 28673 raft_consensus.cc:482] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:55:35.071728 28673 raft_consensus.cc:504] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3473197b94e8476f8143bb534ada9f09" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 40297 } } peers { permanent_uuid: "fed270065b044d3e99692d0663f3aab2" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 40313 } } peers { permanent_uuid: "5bcc077b96d24bf8b37ced1d702717d3" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40341 } }
I20260925 02:55:35.072054 28673 leader_election.cc:288] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3473197b94e8476f8143bb534ada9f09 (127.12.222.195:40297), fed270065b044d3e99692d0663f3aab2 (127.12.222.194:40313)
W20260925 02:55:35.072868 28244 leader_election.cc:341] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fed270065b044d3e99692d0663f3aab2 (127.12.222.194:40313): Not found: Tablet not found: d058255351e543e68aa968a0444cf24b
W20260925 02:55:35.073041 28247 leader_election.cc:341] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3473197b94e8476f8143bb534ada9f09 (127.12.222.195:40297): Not found: Tablet not found: d058255351e543e68aa968a0444cf24b
I20260925 02:55:35.073135 28247 leader_election.cc:302] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5bcc077b96d24bf8b37ced1d702717d3; no voters: 3473197b94e8476f8143bb534ada9f09, fed270065b044d3e99692d0663f3aab2
I20260925 02:55:35.073343 28673 raft_consensus.cc:2726] T d058255351e543e68aa968a0444cf24b P 5bcc077b96d24bf8b37ced1d702717d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:55:35.116057 28674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:35.115716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:40631)
0925 02:55:35.115939 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:35.115986 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
I20260925 02:55:35.845818 28675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:35.845432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:48121)
0925 02:55:35.845680 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:35.845728 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":72,"threads_started":1}
I20260925 02:55:35.956842 28676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:35.956481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:60747)
0925 02:55:35.956708 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:35.956766 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
I20260925 02:55:36.116950 28677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:36.116622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.195:45279)
0925 02:55:36.116828 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:36.116874 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":61,"threads_started":1}
I20260925 02:55:36.847227 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:36.846848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.193:49665)
0925 02:55:36.847068 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:36.847134 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":66,"threads_started":1}
I20260925 02:55:36.957865 28679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:36.957512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:34743 (local address 127.12.222.194:47047)
0925 02:55:36.957740 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:36.957789 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":63,"threads_started":1}
I20260925 02:55:37.104990 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 28194
I20260925 02:55:37.110821 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 28361
I20260925 02:55:37.115685 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 28495
2026-09-25T02:55:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18806 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:55:37 dist-test-slave-252r krb5kdc[28682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:55:37 dist-test-slave-252r krb5kdc[28682](info): set up 2 sockets
Sep 25 02:55:37 dist-test-slave-252r krb5kdc[28682](info): commencing operation
krb5kdc: starting...
W20260925 02:55:38.100140 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:55:38 dist-test-slave-252r krb5kdc[28682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:55:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:55:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:55:38.212224 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:42831
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:42831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:38.312229 28695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:38.312522 28695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:38.312598 28695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:38.316104 28695 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 02:55:38.316171 28695 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 02:55:38.316197 28695 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 02:55:38.316247 28695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:38.316293 28695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:38.316332 28695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:38.316365 28695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:38.316395 28695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:38.321266 28695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:42831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:42831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:38.322148 28695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:38.322993 28695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:55:38.330951 28695 server_base.cc:1034] running on GCE node
W20260925 02:55:38.330957 28701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:38.330900 28700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:38.330881 28703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:38.331593 28695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:38.332594 28695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:38.333770 28695 hybrid_clock.cc:648] HybridClock initialized: now 1790304938333748 us; error 31 us; skew 500 ppm
Sep 25 02:55:38 dist-test-slave-252r krb5kdc[28682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304938, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:38.341912 28695 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:55:38.342651 28695 webserver.cc:460] Webserver started at http://127.12.222.254:33493/ using document root <none> and password file <none>
I20260925 02:55:38.343159 28695 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:38.343212 28695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:38.343415 28695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:38.345188 28695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "1120e5447cea410abdc1c24c44670e3e"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "20f48fb3e5838c1dcd50d4c47d3bd80d"
server_key_iv: "1b2be8ea11bf8392cfade5406c47f104"
server_key_version: "encryptionkey@0"
I20260925 02:55:38.345764 28695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "1120e5447cea410abdc1c24c44670e3e"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "20f48fb3e5838c1dcd50d4c47d3bd80d"
server_key_iv: "1b2be8ea11bf8392cfade5406c47f104"
server_key_version: "encryptionkey@0"
I20260925 02:55:38.349136 28695 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:55:38.351297 28710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:38.352420 28695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:55:38.352530 28695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "1120e5447cea410abdc1c24c44670e3e"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "20f48fb3e5838c1dcd50d4c47d3bd80d"
server_key_iv: "1b2be8ea11bf8392cfade5406c47f104"
server_key_version: "encryptionkey@0"
I20260925 02:55:38.352617 28695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:38.367499 28695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:38.369956 28695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:38.370127 28695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:38.378207 28695 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:42831
I20260925 02:55:38.378536 28762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:42831 every 8 connection(s)
I20260925 02:55:38.379536 28695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:38.383028 28763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:38.387638 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 28695
I20260925 02:55:38.387761 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:55:38.388062 13179 external_mini_cluster.cc:1428] Setting key 0adea599cfa9a637e77afeee5711f227
I20260925 02:55:38.388880 28763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: Bootstrap starting.
I20260925 02:55:38.392278 28763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:38.392913 28763 log.cc:824] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:38.396278 28763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: No bootstrap required, opened a new log
Sep 25 02:55:38 dist-test-slave-252r krb5kdc[28682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:38.400219 28763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1120e5447cea410abdc1c24c44670e3e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42831 } }
I20260925 02:55:38.400503 28763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:38.400563 28763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1120e5447cea410abdc1c24c44670e3e, State: Initialized, Role: FOLLOWER
I20260925 02:55:38.401049 28763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1120e5447cea410abdc1c24c44670e3e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42831 } }
I20260925 02:55:38.401211 28763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:38.401275 28763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:38.401396 28763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:38.402426 28763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1120e5447cea410abdc1c24c44670e3e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42831 } }
I20260925 02:55:38.402748 28763 leader_election.cc:302] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1120e5447cea410abdc1c24c44670e3e; no voters:
I20260925 02:55:38.403047 28763 leader_election.cc:288] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:55:38.403158 28768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:38.403370 28768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 1 LEADER]: Becoming Leader. State: Replica: 1120e5447cea410abdc1c24c44670e3e, State: Running, Role: LEADER
I20260925 02:55:38.403653 28768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1120e5447cea410abdc1c24c44670e3e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42831 } }
I20260925 02:55:38.403838 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:38.389454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48502 (local address 127.12.222.254:42831)
0925 02:55:38.389878 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:38.389896 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:38.389946 (+ 50us) server_negotiation.cc:407] Connection header received
0925 02:55:38.390829 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:38.390850 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:38.391196 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:38.391514 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:38.392282 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:38.393207 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:38.394083 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:38.394478 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:38.394623 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:38.397372 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:38.397377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:38.397390 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:38.397417 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:38.399865 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:38.400592 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:38.400601 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:38.400608 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:38.400773 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:38.401157 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:38.401163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:38.401166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:38.401676 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:38.401880 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:38.402618 (+ 738us) server_negotiation.cc:299] Negotiation successful
0925 02:55:38.402880 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":121,"threads_started":1}
I20260925 02:55:38.404217 28763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:38.405457 28769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1120e5447cea410abdc1c24c44670e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1120e5447cea410abdc1c24c44670e3e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42831 } } }
I20260925 02:55:38.405493 28770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1120e5447cea410abdc1c24c44670e3e. Latest consensus state: current_term: 1 leader_uuid: "1120e5447cea410abdc1c24c44670e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1120e5447cea410abdc1c24c44670e3e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42831 } } }
I20260925 02:55:38.405599 28769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:38.405615 28770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:38.405952 28772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:38.408862 28772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:38.414489 28772 catalog_manager.cc:1347] Generated new cluster ID: 71d2b48eafde4c72b4e48ea6b200e18b
I20260925 02:55:38.414568 28772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:38.425225 28772 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:55:38.426220 28772 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:38.431483 28772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: Generated new TSK 0
I20260925 02:55:38.432126 28772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
I20260925 02:55:38.460860 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:42831
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:38.565142 28789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:38.565375 28789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:38.565442 28789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:38.569088 28789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:38.569151 28789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:38.569248 28789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:55:38.573869 28789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:38.574760 28789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:38.575621 28789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:38.584057 28795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:38.584210 28789 server_base.cc:1034] running on GCE node
W20260925 02:55:38.584264 28797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:38.584374 28794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:38.584731 28789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:38.585228 28789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:38.586372 28789 hybrid_clock.cc:648] HybridClock initialized: now 1790304938586357 us; error 26 us; skew 500 ppm
Sep 25 02:55:38 dist-test-slave-252r krb5kdc[28682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304938, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:38.588860 28789 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:55:38.589596 28789 webserver.cc:460] Webserver started at http://127.12.222.193:43407/ using document root <none> and password file <none>
I20260925 02:55:38.590095 28789 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:38.590178 28789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:38.590416 28789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:38.592167 28789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "e949ce4de3d147de854bbe61e96dc8fd"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "6eca6ee7824474566447bfe162b81ddd"
server_key_iv: "c18348562244faa6b8e0555ab7a33846"
server_key_version: "encryptionkey@0"
I20260925 02:55:38.592721 28789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "e949ce4de3d147de854bbe61e96dc8fd"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "6eca6ee7824474566447bfe162b81ddd"
server_key_iv: "c18348562244faa6b8e0555ab7a33846"
server_key_version: "encryptionkey@0"
I20260925 02:55:38.596002 28789 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:55:38.598225 28804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:38.599301 28789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:55:38.599428 28789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "e949ce4de3d147de854bbe61e96dc8fd"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "6eca6ee7824474566447bfe162b81ddd"
server_key_iv: "c18348562244faa6b8e0555ab7a33846"
server_key_version: "encryptionkey@0"
I20260925 02:55:38.599524 28789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:38.619203 28789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:38.621902 28789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:38.622112 28789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:38.622627 28789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:38.623368 28789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:38.623432 28789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:38.623495 28789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:38.623534 28789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:38.631681 28789 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:44251
I20260925 02:55:38.631739 28917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:44251 every 8 connection(s)
I20260925 02:55:38.632479 28789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:55:38.637838 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 28789
I20260925 02:55:38.638003 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:55:38.638314 13179 external_mini_cluster.cc:1428] Setting key 44e044cda86e5e7c4e6d95cb489237f7
Sep 25 02:55:38 dist-test-slave-252r krb5kdc[28682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304938, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:38.645592 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:38.634045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:48689 (local address 127.12.222.254:42831)
0925 02:55:38.634194 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:38.634199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:38.634993 (+ 794us) server_negotiation.cc:407] Connection header received
0925 02:55:38.636080 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:38.636082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:38.636125 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:38.636193 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:38.637269 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:38.637809 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:38.639209 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:38.639557 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:38.639658 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:38.642142 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:38.642144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:38.642146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:38.642167 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:38.643448 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:38.644053 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:38.644055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:38.644056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:38.644125 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:38.644550 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:38.644552 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:38.644553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:38.644708 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:38.644802 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:38.645399 (+ 597us) server_negotiation.cc:299] Negotiation successful
0925 02:55:38.645488 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:55:38.646627 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:38.634421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:48689)
0925 02:55:38.634847 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:38.634884 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:38.635897 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:38.636383 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:38.636388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:38.636669 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:38.637148 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:38.637162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:38.637943 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:38.637950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:38.639092 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:38.639101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:38.639704 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:38.639708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:38.639867 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:38.640417 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:38.640432 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:38.641960 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:38.643579 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:38.643587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:38.643602 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:38.643942 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:38.644218 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:38.644221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:38.644222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:38.644446 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:38.644820 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:38.644828 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:38.645130 (+ 302us) client_negotiation.cc:769] Sending connection context
0925 02:55:38.645349 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 02:55:38.645668 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":116,"threads_started":1}
I20260925 02:55:38.647737 28918 heartbeater.cc:347] Connected to a master server at 127.12.222.254:42831
WARNING: no policy specified for kudu/127.12.222.194@KRBTEST.COM; defaulting to no policy
I20260925 02:55:38.647967 28918 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:38.648469 28918 heartbeater.cc:511] Master 127.12.222.254:42831 requested a full tablet report, sending...
I20260925 02:55:38.649852 28720 ts_manager.cc:194] Registered new tserver with Master: e949ce4de3d147de854bbe61e96dc8fd (127.12.222.193:44251)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.194@KRBTEST.COM" created.
I20260925 02:55:38.650985 28720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:48689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
I20260925 02:55:38.664588 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:42831
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:38.764451 28923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:38.764668 28923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:38.764705 28923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:38.768350 28923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:38.768392 28923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:38.768477 28923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:55:38.772876 28923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:38.773654 28923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:38.774515 28923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:38.783329 28928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:38.783388 28923 server_base.cc:1034] running on GCE node
W20260925 02:55:38.783429 28929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:38.783327 28931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:38.783931 28923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:38.784574 28923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:38.787853 28923 hybrid_clock.cc:648] HybridClock initialized: now 1790304938787832 us; error 30 us; skew 500 ppm
Sep 25 02:55:38 dist-test-slave-252r krb5kdc[28682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304938, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:38.790452 28923 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:55:38.791173 28923 webserver.cc:460] Webserver started at http://127.12.222.194:35971/ using document root <none> and password file <none>
I20260925 02:55:38.791632 28923 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:38.791685 28923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:38.791895 28923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:38.793710 28923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "b62ca49e40f943a5b442416394149281"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "2a85cee3a76ff2d41208ac988d7aa54a"
server_key_iv: "802bd88d473cd62dc63687ca9ee1d025"
server_key_version: "encryptionkey@0"
I20260925 02:55:38.794256 28923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "b62ca49e40f943a5b442416394149281"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "2a85cee3a76ff2d41208ac988d7aa54a"
server_key_iv: "802bd88d473cd62dc63687ca9ee1d025"
server_key_version: "encryptionkey@0"
I20260925 02:55:38.797577 28923 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 02:55:38.799924 28938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:38.801136 28923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:55:38.801224 28923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "b62ca49e40f943a5b442416394149281"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "2a85cee3a76ff2d41208ac988d7aa54a"
server_key_iv: "802bd88d473cd62dc63687ca9ee1d025"
server_key_version: "encryptionkey@0"
I20260925 02:55:38.801299 28923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:38.815300 28923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:38.818265 28923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:38.818399 28923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:38.818862 28923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:38.819609 28923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:38.819654 28923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:38.819715 28923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:38.819754 28923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:38.827864 28923 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:43343
I20260925 02:55:38.827934 29051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:43343 every 8 connection(s)
I20260925 02:55:38.828701 28923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:55:38.829226 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 28923
I20260925 02:55:38.829360 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:55:38.829669 13179 external_mini_cluster.cc:1428] Setting key 00afe4c98d45d8fe382286b2a7508f60
Sep 25 02:55:38 dist-test-slave-252r krb5kdc[28682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304938, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.222.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.195@KRBTEST.COM" created.
I20260925 02:55:38.841879 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:38.830711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:33757 (local address 127.12.222.254:42831)
0925 02:55:38.830835 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:38.830840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:38.831768 (+ 928us) server_negotiation.cc:407] Connection header received
0925 02:55:38.832981 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:38.832985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:38.833032 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:38.833107 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:38.834175 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:38.834675 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:38.835697 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:38.835911 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:38.836014 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:38.838308 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:38.838311 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:38.838314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:38.838338 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:38.839763 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:38.840272 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:38.840276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:38.840279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:38.840356 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:38.840703 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:38.840707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:38.840708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:38.840961 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:38.841073 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:38.841612 (+ 539us) server_negotiation.cc:299] Negotiation successful
0925 02:55:38.841720 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:55:38.842571 29054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:38.831120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:33757)
0925 02:55:38.831636 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:38.831674 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:38.832788 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:38.833271 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:38.833278 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:38.833613 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:38.834031 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:38.834044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:38.834795 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:38.834798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:38.835572 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:38.835581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:38.836052 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:38.836056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:38.836202 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:38.836778 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:38.836793 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:38.838154 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:38.839877 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:38.839882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:38.839893 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:38.840146 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:38.840459 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:38.840462 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:38.840464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:38.840619 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:38.841074 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:38.841081 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:38.841380 (+ 299us) client_negotiation.cc:769] Sending connection context
0925 02:55:38.841570 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 02:55:38.841848 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":160,"threads_started":1}
I20260925 02:55:38.843668 29052 heartbeater.cc:347] Connected to a master server at 127.12.222.254:42831
I20260925 02:55:38.843904 29052 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:38.844429 29052 heartbeater.cc:511] Master 127.12.222.254:42831 requested a full tablet report, sending...
I20260925 02:55:38.845520 28720 ts_manager.cc:194] Registered new tserver with Master: b62ca49e40f943a5b442416394149281 (127.12.222.194:43343)
I20260925 02:55:38.846359 28720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:33757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
I20260925 02:55:38.856772 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:42831
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:38.959677 29057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:38.959872 29057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:38.959913 29057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:38.963552 29057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:38.963596 29057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:38.963661 29057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:55:38.968189 29057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:38.969010 29057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:38.969789 29057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:38.978404 29066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:38.978508 29057 server_base.cc:1034] running on GCE node
W20260925 02:55:38.978389 29062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:38.978420 29064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:38.979005 29057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:38.979562 29057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:38.980734 29057 hybrid_clock.cc:648] HybridClock initialized: now 1790304938980720 us; error 30 us; skew 500 ppm
Sep 25 02:55:38 dist-test-slave-252r krb5kdc[28682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304938, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:38.983177 29057 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:55:38.983870 29057 webserver.cc:460] Webserver started at http://127.12.222.195:45237/ using document root <none> and password file <none>
I20260925 02:55:38.984387 29057 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:38.984472 29057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:38.984665 29057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:55:38.986333 29057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "747fcb3d278946acb50e1b79daf99ca9"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "875cd534352d9c71a8c4d6d2ebe01b7c"
server_key_iv: "1a6fa3d1ac6d220109dfde4f8ac39b3c"
server_key_version: "encryptionkey@0"
I20260925 02:55:38.986826 29057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "747fcb3d278946acb50e1b79daf99ca9"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "875cd534352d9c71a8c4d6d2ebe01b7c"
server_key_iv: "1a6fa3d1ac6d220109dfde4f8ac39b3c"
server_key_version: "encryptionkey@0"
I20260925 02:55:38.990124 29057 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:55:38.992303 29073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:38.993330 29057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:55:38.993448 29057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "747fcb3d278946acb50e1b79daf99ca9"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "875cd534352d9c71a8c4d6d2ebe01b7c"
server_key_iv: "1a6fa3d1ac6d220109dfde4f8ac39b3c"
server_key_version: "encryptionkey@0"
I20260925 02:55:38.993541 29057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:39.011766 29057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:39.014676 29057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:39.014851 29057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:39.015341 29057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:39.016038 29057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:39.016103 29057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:39.016165 29057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:39.016211 29057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:39.024214 29057 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:36371
I20260925 02:55:39.024302 29187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:36371 every 8 connection(s)
I20260925 02:55:39.025032 29057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:55:39.031735 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 29057
I20260925 02:55:39.031874 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:55:39.032166 13179 external_mini_cluster.cc:1428] Setting key ad76ff1e1f07b65b82eefcf8c1ca3156
Sep 25 02:55:39 dist-test-slave-252r krb5kdc[28682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304938, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:39.036995 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:39.026575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:45179 (local address 127.12.222.254:42831)
0925 02:55:39.026702 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:39.026706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:39.027398 (+ 692us) server_negotiation.cc:407] Connection header received
0925 02:55:39.028633 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:39.028637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:39.028693 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:39.028774 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:39.029772 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.030264 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.031281 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.031592 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.031705 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:39.034329 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:39.034332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:39.034334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:39.034355 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:39.035401 (+ 1046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:39.035814 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:39.035817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:39.035818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:39.035883 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:39.036141 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:39.036144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:39.036145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:39.036347 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:39.036430 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:39.036803 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 02:55:39.036894 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:55:39.037755 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:39.026869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:45179)
0925 02:55:39.027259 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:39.027295 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:39.028433 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:39.028956 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:39.028962 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:39.029254 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:39.029635 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:39.029646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.030391 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.030399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:39.031162 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:39.031170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.031730 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.031733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:39.031951 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:39.032542 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:39.032557 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:39.034198 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:39.035491 (+ 1293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:39.035494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:39.035503 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:39.035735 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:39.035956 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:39.035958 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:39.035959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:39.036071 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:39.036435 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:39.036439 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:39.036624 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 02:55:39.036758 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 02:55:39.036960 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":107,"threads_started":1}
I20260925 02:55:39.038705 29188 heartbeater.cc:347] Connected to a master server at 127.12.222.254:42831
I20260925 02:55:39.038944 29188 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:39.039367 29188 heartbeater.cc:511] Master 127.12.222.254:42831 requested a full tablet report, sending...
I20260925 02:55:39.040305 28720 ts_manager.cc:194] Registered new tserver with Master: 747fcb3d278946acb50e1b79daf99ca9 (127.12.222.195:36371)
I20260925 02:55:39.040812 28720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:45179
I20260925 02:55:39.045500 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:55:39.053505 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:39.047291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.12.222.254:42831)
0925 02:55:39.047434 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:39.047437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:39.047599 (+ 162us) server_negotiation.cc:407] Connection header received
0925 02:55:39.047697 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:39.047699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:39.047751 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:39.047807 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:39.048446 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.048894 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.049600 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.049779 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.049843 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:39.050862 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:39.050864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:39.050866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:39.050883 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:39.052184 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:39.052595 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:39.052597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:39.052598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:39.052661 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:39.052865 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:39.052867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:39.052868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:39.053038 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:39.053133 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:39.053271 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 02:55:39.053344 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:55:39.055764 28720 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:55:39.058619 28720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:55:39.071707 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:39.066060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.12.222.195:36371)
0925 02:55:39.067192 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:39.067203 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:39.067224 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:55:39.067286 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:39.067291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:39.067478 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:39.067654 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:39.067886 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.068823 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.070353 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.071065 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.071195 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:39.071312 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:39.071484 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 02:55:39.071567 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1026,"thread_start_us":81,"threads_started":1}
I20260925 02:55:39.071691 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:39.065965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:36371 (local address 127.0.0.1:51674)
0925 02:55:39.066453 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:39.066496 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:39.066625 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:39.067642 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:39.067645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:39.067663 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:39.067758 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:39.067766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.068956 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.068960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:39.070243 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:39.070254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.071184 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.071187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:39.071289 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:39.071323 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 02:55:39.071454 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 02:55:39.071567 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":68,"threads_started":1}
I20260925 02:55:39.072012 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:39.066732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41550 (local address 127.12.222.194:43343)
0925 02:55:39.067098 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:39.067111 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:39.067136 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:55:39.067216 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:39.067220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:39.067400 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:39.067548 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:39.067829 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.069029 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.070453 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.071104 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.071215 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:39.071319 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:39.071762 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 02:55:39.071875 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":107,"threads_started":1}
I20260925 02:55:39.072098 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:39.066077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:43343 (local address 127.0.0.1:41550)
0925 02:55:39.066724 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:39.066737 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:39.066828 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:39.067536 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:39.067542 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:39.067562 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:39.067694 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:39.067702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.069158 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.069162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:39.070336 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:39.070344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.071382 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.071386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:39.071485 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:39.071508 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:55:39.071579 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 02:55:39.071648 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":7,"thread_start_us":83,"threads_started":1}
I20260925 02:55:39.072758 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:39.066346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:44251 (local address 127.0.0.1:32966)
0925 02:55:39.067874 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:39.067886 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:39.067981 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:39.068427 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:39.068430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:39.068448 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:39.068522 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:39.068528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.069628 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.069634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:39.070722 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:39.070733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.072380 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.072385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:39.072476 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:39.072499 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:55:39.072582 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 02:55:39.072645 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1464,"thread_start_us":90,"threads_started":1}
I20260925 02:55:39.072850 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:39.066720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.12.222.193:44251)
0925 02:55:39.067853 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:39.067865 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:39.067974 (+ 109us) server_negotiation.cc:407] Connection header received
0925 02:55:39.068082 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:39.068087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:39.068288 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:39.068423 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:39.068641 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.069505 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.070851 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.071638 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.072285 (+ 647us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:39.072419 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:39.072625 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 02:55:39.072717 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1039,"thread_start_us":101,"threads_started":1}
I20260925 02:55:39.074539 29121 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4fa80322ef4a9898e7f748cd85a2ca (DEFAULT_TABLE table=test-table [id=d0be1e4380cc4d2e87027dd52faec3f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:39.074783 28986 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4fa80322ef4a9898e7f748cd85a2ca (DEFAULT_TABLE table=test-table [id=d0be1e4380cc4d2e87027dd52faec3f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:39.075486 29121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4fa80322ef4a9898e7f748cd85a2ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:39.075778 28986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4fa80322ef4a9898e7f748cd85a2ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:39.075448 28852 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4fa80322ef4a9898e7f748cd85a2ca (DEFAULT_TABLE table=test-table [id=d0be1e4380cc4d2e87027dd52faec3f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:39.076387 28852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4fa80322ef4a9898e7f748cd85a2ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:39.081459 29206 tablet_bootstrap.cc:492] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9: Bootstrap starting.
I20260925 02:55:39.082823 29208 tablet_bootstrap.cc:492] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd: Bootstrap starting.
I20260925 02:55:39.083288 29207 tablet_bootstrap.cc:492] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281: Bootstrap starting.
I20260925 02:55:39.084091 29206 tablet_bootstrap.cc:654] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:39.085078 29206 log.cc:824] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:39.085196 29208 tablet_bootstrap.cc:654] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:39.085242 29207 tablet_bootstrap.cc:654] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:39.085991 29208 log.cc:824] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:39.086050 29207 log.cc:824] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:39.087033 29206 tablet_bootstrap.cc:492] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9: No bootstrap required, opened a new log
I20260925 02:55:39.087212 29206 ts_tablet_manager.cc:1397] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:55:39.087668 29208 tablet_bootstrap.cc:492] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd: No bootstrap required, opened a new log
I20260925 02:55:39.087879 29208 ts_tablet_manager.cc:1397] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:55:39.088088 29207 tablet_bootstrap.cc:492] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281: No bootstrap required, opened a new log
I20260925 02:55:39.088280 29207 ts_tablet_manager.cc:1397] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 02:55:39.090588 29208 raft_consensus.cc:348] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:39.090838 29208 raft_consensus.cc:374] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:39.090905 29208 raft_consensus.cc:729] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e949ce4de3d147de854bbe61e96dc8fd, State: Initialized, Role: FOLLOWER
I20260925 02:55:39.090849 29206 raft_consensus.cc:348] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:39.091080 29206 raft_consensus.cc:374] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:39.091123 29206 raft_consensus.cc:729] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747fcb3d278946acb50e1b79daf99ca9, State: Initialized, Role: FOLLOWER
I20260925 02:55:39.091343 29208 consensus_queue.cc:260] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:39.091492 29206 consensus_queue.cc:260] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:39.091709 29207 raft_consensus.cc:348] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:39.091938 29207 raft_consensus.cc:374] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:39.091990 29207 raft_consensus.cc:729] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b62ca49e40f943a5b442416394149281, State: Initialized, Role: FOLLOWER
I20260925 02:55:39.092185 29188 heartbeater.cc:503] Master 127.12.222.254:42831 was elected leader, sending a full tablet report...
I20260925 02:55:39.092414 29206 ts_tablet_manager.cc:1428] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 02:55:39.092360 29207 consensus_queue.cc:260] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:39.092685 28918 heartbeater.cc:503] Master 127.12.222.254:42831 was elected leader, sending a full tablet report...
I20260925 02:55:39.092979 29052 heartbeater.cc:503] Master 127.12.222.254:42831 was elected leader, sending a full tablet report...
I20260925 02:55:39.093221 29207 ts_tablet_manager.cc:1428] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:55:39.093842 29208 ts_tablet_manager.cc:1428] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 02:55:39.112705 29214 raft_consensus.cc:482] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:55:39.112829 29214 raft_consensus.cc:504] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:39.113626 29214 leader_election.cc:288] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 747fcb3d278946acb50e1b79daf99ca9 (127.12.222.195:36371), e949ce4de3d147de854bbe61e96dc8fd (127.12.222.193:44251)
I20260925 02:55:39.117097 29054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:39.113821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:36371 (local address 127.12.222.194:38819)
0925 02:55:39.113943 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:39.113959 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:39.114041 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:39.114304 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:39.114318 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:39.114339 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:39.114454 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:39.114461 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.115124 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.115129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:39.116024 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:39.116032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.116704 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.116707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:39.116802 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:39.116833 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 02:55:39.116899 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 02:55:39.116971 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":26}
I20260925 02:55:39.117229 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:39.113901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:38819 (local address 127.12.222.195:36371)
0925 02:55:39.114008 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:39.114013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:39.114028 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:55:39.114152 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:39.114156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:39.114196 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:39.114278 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:39.114558 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.115015 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.116146 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.116584 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.116712 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:39.116766 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:39.117048 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 02:55:39.117119 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 02:55:39.117286 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:39.113929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:48083 (local address 127.12.222.193:44251)
0925 02:55:39.114027 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:39.114031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:39.114397 (+ 366us) server_negotiation.cc:407] Connection header received
0925 02:55:39.114463 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:39.114465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:39.114495 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:39.114573 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:39.114806 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.115225 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.116428 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.116782 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.116900 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:39.116946 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:39.117132 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 02:55:39.117191 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 02:55:39.117304 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:39.113821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:44251 (local address 127.12.222.194:48083)
0925 02:55:39.114304 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:39.114316 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:39.114395 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:39.114589 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:39.114592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:39.114606 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:39.114710 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:39.114715 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.115332 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.115335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:39.116314 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:39.116323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.116900 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.116904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:39.117003 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:39.117022 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 02:55:39.117116 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:55:39.117177 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":187,"thread_start_us":128,"threads_started":1}
I20260925 02:55:39.117851 28872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4fa80322ef4a9898e7f748cd85a2ca" candidate_uuid: "b62ca49e40f943a5b442416394149281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e949ce4de3d147de854bbe61e96dc8fd" is_pre_election: true
I20260925 02:55:39.117928 29142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4fa80322ef4a9898e7f748cd85a2ca" candidate_uuid: "b62ca49e40f943a5b442416394149281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747fcb3d278946acb50e1b79daf99ca9" is_pre_election: true
I20260925 02:55:39.118108 28872 raft_consensus.cc:2454] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b62ca49e40f943a5b442416394149281 in term 0.
I20260925 02:55:39.118114 29142 raft_consensus.cc:2454] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b62ca49e40f943a5b442416394149281 in term 0.
I20260925 02:55:39.118574 28939 leader_election.cc:302] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 747fcb3d278946acb50e1b79daf99ca9, b62ca49e40f943a5b442416394149281, e949ce4de3d147de854bbe61e96dc8fd; no voters:
I20260925 02:55:39.118882 29214 raft_consensus.cc:2781] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:55:39.118985 29214 raft_consensus.cc:482] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:55:39.119043 29214 raft_consensus.cc:3037] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:39.119911 29214 raft_consensus.cc:504] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:39.120215 29214 leader_election.cc:288] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [CANDIDATE]: Term 1 election: Requested vote from peers 747fcb3d278946acb50e1b79daf99ca9 (127.12.222.195:36371), e949ce4de3d147de854bbe61e96dc8fd (127.12.222.193:44251)
I20260925 02:55:39.120543 28872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4fa80322ef4a9898e7f748cd85a2ca" candidate_uuid: "b62ca49e40f943a5b442416394149281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e949ce4de3d147de854bbe61e96dc8fd"
I20260925 02:55:39.120543 29142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4fa80322ef4a9898e7f748cd85a2ca" candidate_uuid: "b62ca49e40f943a5b442416394149281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747fcb3d278946acb50e1b79daf99ca9"
I20260925 02:55:39.120648 28872 raft_consensus.cc:3037] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:39.120648 29142 raft_consensus.cc:3037] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:39.121794 29142 raft_consensus.cc:2454] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b62ca49e40f943a5b442416394149281 in term 1.
I20260925 02:55:39.121794 28872 raft_consensus.cc:2454] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b62ca49e40f943a5b442416394149281 in term 1.
I20260925 02:55:39.122078 28939 leader_election.cc:302] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b62ca49e40f943a5b442416394149281, e949ce4de3d147de854bbe61e96dc8fd; no voters:
I20260925 02:55:39.122241 29214 raft_consensus.cc:2781] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:55:39.122422 29214 raft_consensus.cc:686] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [term 1 LEADER]: Becoming Leader. State: Replica: b62ca49e40f943a5b442416394149281, State: Running, Role: LEADER
I20260925 02:55:39.122686 29214 consensus_queue.cc:237] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:39.125419 28724 catalog_manager.cc:5495] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 reported cstate change: term changed from 0 to 1, leader changed from <none> to b62ca49e40f943a5b442416394149281 (127.12.222.194). New cstate: current_term: 1 leader_uuid: "b62ca49e40f943a5b442416394149281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } health_report { overall_health: UNKNOWN } } }
W20260925 02:55:39.133659 28919 tablet.cc:2369] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:55:39.136294 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:39.133120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.12.222.194:43343)
0925 02:55:39.133216 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:39.133222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:39.133261 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:55:39.133454 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:39.133457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:39.133498 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:39.133595 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:55:39.133841 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.134387 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.135122 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:39.135297 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:39.135360 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:39.135585 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:55:39.135901 (+ 316us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:55:39.135980 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:39.136048 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 02:55:39.136087 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:55:39.141062 29142 raft_consensus.cc:1261] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9 [term 1 FOLLOWER]: Refusing update from remote peer b62ca49e40f943a5b442416394149281: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:39.141064 28872 raft_consensus.cc:1261] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd [term 1 FOLLOWER]: Refusing update from remote peer b62ca49e40f943a5b442416394149281: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:55:39.141654 29216 consensus_queue.cc:1035] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:39.141835 29214 consensus_queue.cc:1035] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:55:39.146914 29219 mvcc.cc:205] Tried to move back new op lower bound from 7333089030716248064 to 7333089030648971264. Current Snapshot: MvccSnapshot[applied={T|T < 7333089030716248064}]
I20260925 02:55:39.147796 29221 mvcc.cc:205] Tried to move back new op lower bound from 7333089030716248064 to 7333089030648971264. Current Snapshot: MvccSnapshot[applied={T|T < 7333089030716248064}]
I20260925 02:55:39.155642 28727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:39.155857 28727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:55:39.158179 28727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: Sending DeleteTablet for 3 replicas of tablet 7e4fa80322ef4a9898e7f748cd85a2ca
I20260925 02:55:39.158830 29121 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4fa80322ef4a9898e7f748cd85a2ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:39 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:51674
I20260925 02:55:39.158906 28986 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4fa80322ef4a9898e7f748cd85a2ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:39 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:41550
I20260925 02:55:39.159026 28852 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4fa80322ef4a9898e7f748cd85a2ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:39 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:32966
I20260925 02:55:39.159341 29230 tablet_replica.cc:330] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281: stopping tablet replica
I20260925 02:55:39.159416 29231 tablet_replica.cc:330] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd: stopping tablet replica
I20260925 02:55:39.159582 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 28789
I20260925 02:55:39.159608 29229 tablet_replica.cc:330] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9: stopping tablet replica
I20260925 02:55:39.159746 29230 raft_consensus.cc:2229] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:55:39.159857 29229 raft_consensus.cc:2229] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:39.160123 29229 raft_consensus.cc:2258] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:39.160344 29230 raft_consensus.cc:2258] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:39.161792 29230 ts_tablet_manager.cc:1905] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:39.164474 29230 ts_tablet_manager.cc:1918] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:39.164618 29230 log.cc:1196] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/7e4fa80322ef4a9898e7f748cd85a2ca
I20260925 02:55:39.165108 29230 ts_tablet_manager.cc:1939] T 7e4fa80322ef4a9898e7f748cd85a2ca P b62ca49e40f943a5b442416394149281: Deleting consensus metadata
I20260925 02:55:39.165529 29229 ts_tablet_manager.cc:1905] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:39.166261 28713 catalog_manager.cc:4841] TS b62ca49e40f943a5b442416394149281 (127.12.222.194:43343): tablet 7e4fa80322ef4a9898e7f748cd85a2ca (table test-table [id=d0be1e4380cc4d2e87027dd52faec3f6]) successfully deleted
W20260925 02:55:39.167249 28711 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.222.193:44251 (error 108)
I20260925 02:55:39.167477 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 28923
I20260925 02:55:39.169098 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:39.168732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:44251 (local address 127.0.0.1:32968)
0925 02:55:39.168978 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:39.169034 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.193:44251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
W20260925 02:55:39.169315 28711 catalog_manager.cc:4584] TS e949ce4de3d147de854bbe61e96dc8fd (127.12.222.193:44251): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e4fa80322ef4a9898e7f748cd85a2ca: Network error: Client connection negotiation failed: client connection to 127.12.222.193:44251: connect: Connection refused (error 111)
I20260925 02:55:39.170079 29229 ts_tablet_manager.cc:1918] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:39.170187 29229 log.cc:1196] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/7e4fa80322ef4a9898e7f748cd85a2ca
I20260925 02:55:39.170504 29229 ts_tablet_manager.cc:1939] T 7e4fa80322ef4a9898e7f748cd85a2ca P 747fcb3d278946acb50e1b79daf99ca9: Deleting consensus metadata
I20260925 02:55:39.171455 28712 catalog_manager.cc:4841] TS 747fcb3d278946acb50e1b79daf99ca9 (127.12.222.195:36371): tablet 7e4fa80322ef4a9898e7f748cd85a2ca (table test-table [id=d0be1e4380cc4d2e87027dd52faec3f6]) successfully deleted
I20260925 02:55:39.173557 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 29057
I20260925 02:55:39.179929 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 28695
I20260925 02:55:59.186518 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:42831
--webserver_interface=127.12.222.254
--webserver_port=33493
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:42831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:55:59.288632 29233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:59.288828 29233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:59.288862 29233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:59.292763 29233 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 02:55:59.292802 29233 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 02:55:59.292817 29233 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 02:55:59.292830 29233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:55:59.292887 29233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:59.292922 29233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:59.292959 29233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:55:59.293017 29233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:55:59.297784 29233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:42831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:42831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=33493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:59.298588 29233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:59.299418 29233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:59.311131 29239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:59.311266 29240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:59.311292 29233 server_base.cc:1034] running on GCE node
W20260925 02:55:59.311154 29242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:59.311995 29233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:59.313088 29233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:59.314270 29233 hybrid_clock.cc:648] HybridClock initialized: now 1790304959314257 us; error 22 us; skew 500 ppm
Sep 25 02:55:59 dist-test-slave-252r krb5kdc[28682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304959, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:59.316857 29233 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:55:59.317704 29233 webserver.cc:460] Webserver started at http://127.12.222.254:33493/ using document root <none> and password file <none>
I20260925 02:55:59.318235 29233 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:59.318341 29233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:59.321983 29233 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:55:59.323994 29249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:59.325155 29233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:55:59.325292 29233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "1120e5447cea410abdc1c24c44670e3e"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "20f48fb3e5838c1dcd50d4c47d3bd80d"
server_key_iv: "1b2be8ea11bf8392cfade5406c47f104"
server_key_version: "encryptionkey@0"
I20260925 02:55:59.325847 29233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:59.371922 29233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:59.374504 29233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:59.374687 29233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:59.381201 29233 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:42831
I20260925 02:55:59.381469 29302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:42831 every 8 connection(s)
I20260925 02:55:59.382210 29233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:55:59.382743 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 29233
I20260925 02:55:59.383183 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:44251
--local_ip_for_outbound_sockets=127.12.222.193
--tserver_master_addrs=127.12.222.254:42831
--webserver_port=43407
--webserver_interface=127.12.222.193
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:55:59.386480 29303 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:55:59.388029 29303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: Bootstrap starting.
I20260925 02:55:59.403720 29303 log.cc:824] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:59.411443 29303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:55:59.411765 29303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: Bootstrap complete.
I20260925 02:55:59.415449 29303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1120e5447cea410abdc1c24c44670e3e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42831 } }
I20260925 02:55:59.415820 29303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1120e5447cea410abdc1c24c44670e3e, State: Initialized, Role: FOLLOWER
I20260925 02:55:59.416272 29303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1120e5447cea410abdc1c24c44670e3e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42831 } }
I20260925 02:55:59.416409 29303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:55:59.416491 29303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:55:59.416594 29303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:55:59.418298 29303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1120e5447cea410abdc1c24c44670e3e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42831 } }
I20260925 02:55:59.418617 29303 leader_election.cc:302] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1120e5447cea410abdc1c24c44670e3e; no voters:
I20260925 02:55:59.418897 29303 leader_election.cc:288] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:55:59.419010 29307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:55:59.419232 29307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 2 LEADER]: Becoming Leader. State: Replica: 1120e5447cea410abdc1c24c44670e3e, State: Running, Role: LEADER
I20260925 02:55:59.419512 29307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1120e5447cea410abdc1c24c44670e3e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42831 } }
I20260925 02:55:59.420013 29303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:55:59.420473 29309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1120e5447cea410abdc1c24c44670e3e. Latest consensus state: current_term: 2 leader_uuid: "1120e5447cea410abdc1c24c44670e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1120e5447cea410abdc1c24c44670e3e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42831 } } }
I20260925 02:55:59.420585 29309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:59.420835 29308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1120e5447cea410abdc1c24c44670e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1120e5447cea410abdc1c24c44670e3e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42831 } } }
I20260925 02:55:59.420945 29308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [sys.catalog]: This master's current role is: LEADER
I20260925 02:55:59.425009 29323 catalog_manager.cc:1259] Loaded cluster ID: 71d2b48eafde4c72b4e48ea6b200e18b
I20260925 02:55:59.425065 29323 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: loading cluster ID for follower catalog manager: success
I20260925 02:55:59.427356 29323 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: acquiring CA information for follower catalog manager: success
I20260925 02:55:59.428791 29323 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:55:59.428915 29311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:55:59.431174 29311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:55:59.431653 29311 catalog_manager.cc:1259] Loaded cluster ID: 71d2b48eafde4c72b4e48ea6b200e18b
I20260925 02:55:59.431710 29311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:55:59.432513 29311 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:55:59.432976 29311 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: Loaded TSK: 0
I20260925 02:55:59.451166 29311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: Generated new TSK 1
I20260925 02:55:59.451370 29311 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:55:59.488763 29305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:59.488986 29305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:59.489070 29305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:59.492969 29305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:59.493010 29305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:59.493086 29305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:55:59.497789 29305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:44251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=43407
--tserver_master_addrs=127.12.222.254:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:59.498615 29305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:59.499573 29305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:59.507905 29331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:59.507745 29330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:59.507730 29333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:59.508366 29305 server_base.cc:1034] running on GCE node
I20260925 02:55:59.508729 29305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:59.509361 29305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:59.510550 29305 hybrid_clock.cc:648] HybridClock initialized: now 1790304959510523 us; error 35 us; skew 500 ppm
Sep 25 02:55:59 dist-test-slave-252r krb5kdc[28682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304959, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:59.513586 29305 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:55:59.514370 29305 webserver.cc:460] Webserver started at http://127.12.222.193:43407/ using document root <none> and password file <none>
I20260925 02:55:59.514837 29305 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:59.514947 29305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:59.518296 29305 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:55:59.520397 29340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:59.521638 29305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:55:59.521742 29305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "e949ce4de3d147de854bbe61e96dc8fd"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "6eca6ee7824474566447bfe162b81ddd"
server_key_iv: "c18348562244faa6b8e0555ab7a33846"
server_key_version: "encryptionkey@0"
I20260925 02:55:59.522289 29305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:59.557106 29305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:59.560456 29305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:59.560702 29305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:59.561353 29305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:59.562352 29347 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 02:55:59.565943 29305 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:55:59.566007 29305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260925 02:55:59.566078 29305 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:55:59.567548 29305 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:55:59.567615 29305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 02:55:59.567835 29347 tablet_bootstrap.cc:492] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd: Bootstrap starting.
I20260925 02:55:59.577518 29305 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:44251
I20260925 02:55:59.577622 29456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:44251 every 8 connection(s)
I20260925 02:55:59.578368 29305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:55:59.580361 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 29305
I20260925 02:55:59.580906 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:43343
--local_ip_for_outbound_sockets=127.12.222.194
--tserver_master_addrs=127.12.222.254:42831
--webserver_port=35971
--webserver_interface=127.12.222.194
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:55:59.585301 29347 log.cc:824] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:55:59 dist-test-slave-252r krb5kdc[28682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304959, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:59.594192 29347 tablet_bootstrap.cc:492] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:55:59.594458 29460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:59.580332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:46463)
0925 02:55:59.581467 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:59.581522 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:59.582594 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:59.583591 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:59.583598 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:59.583966 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:59.584448 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:59.584463 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.585879 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.585882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:59.586776 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:59.586785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.587503 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.587506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:59.587695 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:59.588213 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:59.588267 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:59.589661 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:59.591675 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:59.591680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:59.591690 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:59.591934 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:59.592293 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:59.592296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:59.592298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:59.592426 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:59.592951 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:59.592956 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:59.593166 (+ 210us) client_negotiation.cc:769] Sending connection context
0925 02:55:59.593325 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 02:55:59.593607 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":166,"threads_started":1}
I20260925 02:55:59.594447 29459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:59.580310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:46463 (local address 127.12.222.254:42831)
0925 02:55:59.580845 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:59.580863 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:59.581658 (+ 795us) server_negotiation.cc:407] Connection header received
0925 02:55:59.582971 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:59.582981 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:59.583235 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:59.583539 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:59.584613 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.585717 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.586926 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.587352 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.587529 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:59.589853 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:59.589857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:59.589868 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:59.589895 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:59.591558 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:59.592067 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:59.592071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:59.592074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:59.592175 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:59.592524 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:59.592527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:59.592528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:59.592821 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:59.592970 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:59.593452 (+ 482us) server_negotiation.cc:299] Negotiation successful
0925 02:55:59.593680 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":114,"threads_started":1}
I20260925 02:55:59.594568 29347 tablet_bootstrap.cc:492] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd: Bootstrap complete.
I20260925 02:55:59.595858 29347 ts_tablet_manager.cc:1397] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.004s
I20260925 02:55:59.596879 29457 heartbeater.cc:347] Connected to a master server at 127.12.222.254:42831
I20260925 02:55:59.597067 29457 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:59.597469 29457 heartbeater.cc:511] Master 127.12.222.254:42831 requested a full tablet report, sending...
I20260925 02:55:59.598390 29347 raft_consensus.cc:348] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:59.598834 29347 raft_consensus.cc:729] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e949ce4de3d147de854bbe61e96dc8fd, State: Initialized, Role: FOLLOWER
I20260925 02:55:59.599260 29347 consensus_queue.cc:260] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:59.599664 29266 ts_manager.cc:194] Registered new tserver with Master: e949ce4de3d147de854bbe61e96dc8fd (127.12.222.193:44251)
I20260925 02:55:59.600157 29347 ts_tablet_manager.cc:1428] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20260925 02:55:59.602834 29266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:46463
I20260925 02:55:59.605207 29457 heartbeater.cc:503] Master 127.12.222.254:42831 was elected leader, sending a full tablet report...
Sep 25 02:55:59 dist-test-slave-252r krb5kdc[28682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304959, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for kudu/127.12.222.193@KRBTEST.COM
I20260925 02:55:59.611688 29464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:59.602231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:44251 (local address 127.0.0.1:36900)
0925 02:55:59.602868 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:59.602922 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:59.603087 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:59.603559 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:59.603565 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:59.603923 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:59.604025 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:59.604032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.605255 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.605260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:59.605986 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:59.605994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.606669 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.606672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:59.606764 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:59.607164 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:59.607176 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:59.608458 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:59.610291 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:59.610299 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:59.610302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:59.610517 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:59.610813 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:59.610814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:59.610815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:59.610878 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:59.611284 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:59.611287 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:59.611363 (+ 76us) client_negotiation.cc:769] Sending connection context
0925 02:55:59.611458 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 02:55:59.611586 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":85,"threads_started":1}
I20260925 02:55:59.611843 29465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:59.602331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36900 (local address 127.12.222.193:44251)
0925 02:55:59.602636 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:59.602646 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:59.602998 (+ 352us) server_negotiation.cc:407] Connection header received
0925 02:55:59.603235 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:59.603240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:59.603415 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:59.603548 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:59.604148 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.605118 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.606149 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.606467 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.606569 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:59.608586 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:59.608594 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:59.608603 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:59.608625 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:59.610178 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:59.610620 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:59.610626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:59.610627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:59.610715 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:59.610954 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:59.610956 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:59.610958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:59.611186 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:59.611299 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:59.611567 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 02:55:59.611706 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":87,"threads_started":1}
I20260925 02:55:59.612637 29389 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4fa80322ef4a9898e7f748cd85a2ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:55:39 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:36900
I20260925 02:55:59.613289 29466 tablet_replica.cc:330] stopping tablet replica
I20260925 02:55:59.613529 29466 raft_consensus.cc:2229] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:55:59.613749 29466 raft_consensus.cc:2258] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:55:59.614543 29466 ts_tablet_manager.cc:1905] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:55:59.618091 29466 ts_tablet_manager.cc:1918] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:55:59.618192 29466 log.cc:1196] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/wal/wals/7e4fa80322ef4a9898e7f748cd85a2ca
I20260925 02:55:59.618450 29466 ts_tablet_manager.cc:1939] T 7e4fa80322ef4a9898e7f748cd85a2ca P e949ce4de3d147de854bbe61e96dc8fd: Deleting consensus metadata
I20260925 02:55:59.619784 29250 catalog_manager.cc:4841] TS e949ce4de3d147de854bbe61e96dc8fd (127.12.222.193:44251): tablet 7e4fa80322ef4a9898e7f748cd85a2ca (table test-table [id=d0be1e4380cc4d2e87027dd52faec3f6]) successfully deleted
W20260925 02:55:59.694129 29461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:59.694365 29461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:59.694438 29461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:59.698289 29461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:59.698365 29461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:59.698446 29461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:55:59.703368 29461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:43343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=35971
--tserver_master_addrs=127.12.222.254:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:59.704285 29461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:59.705142 29461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:59.713579 29472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:59.713620 29471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:59.713620 29474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:59.713757 29461 server_base.cc:1034] running on GCE node
I20260925 02:55:59.714200 29461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:59.714766 29461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:59.715946 29461 hybrid_clock.cc:648] HybridClock initialized: now 1790304959715935 us; error 22 us; skew 500 ppm
Sep 25 02:55:59 dist-test-slave-252r krb5kdc[28682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304959, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:59.718413 29461 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:55:59.719129 29461 webserver.cc:460] Webserver started at http://127.12.222.194:35971/ using document root <none> and password file <none>
I20260925 02:55:59.719717 29461 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:59.719815 29461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:59.723394 29461 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:55:59.725172 29481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:59.726301 29461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:55:59.726418 29461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "b62ca49e40f943a5b442416394149281"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "2a85cee3a76ff2d41208ac988d7aa54a"
server_key_iv: "802bd88d473cd62dc63687ca9ee1d025"
server_key_version: "encryptionkey@0"
I20260925 02:55:59.726861 29461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:59.743129 29461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:59.745548 29461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:59.745765 29461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:59.746325 29461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:59.747120 29461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:59.747188 29461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:59.747249 29461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:59.747296 29461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:59.755620 29461 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:43343
I20260925 02:55:59.755642 29596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:43343 every 8 connection(s)
I20260925 02:55:59.756567 29461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:55:59.757958 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 29461
I20260925 02:55:59.758428 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:36371
--local_ip_for_outbound_sockets=127.12.222.195
--tserver_master_addrs=127.12.222.254:42831
--webserver_port=45237
--webserver_interface=127.12.222.195
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:55:59 dist-test-slave-252r krb5kdc[28682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304959, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:59.769265 29459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:59.758110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:55765 (local address 127.12.222.254:42831)
0925 02:55:59.758277 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:59.758283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:59.759054 (+ 771us) server_negotiation.cc:407] Connection header received
0925 02:55:59.760280 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:59.760284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:59.760348 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:59.760452 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:59.761507 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.762073 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.763102 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.763347 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.763452 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:59.766027 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:59.766030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:59.766032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:59.766054 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:59.767346 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:59.767910 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:59.767913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:59.767914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:59.767976 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:59.768349 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:59.768351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:59.768352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:59.768501 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:59.768568 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:59.769049 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 02:55:59.769158 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:55:59.770153 29599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:59.758444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:55765)
0925 02:55:59.758895 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:59.758939 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:59.760057 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:59.760621 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:59.760627 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:59.760950 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:59.761352 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:59.761364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.762192 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.762195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:59.762976 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:59.762985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.763489 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.763493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:59.763643 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:59.764163 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:59.764175 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:59.765856 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:59.767457 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:59.767464 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:59.767477 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:59.767808 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:59.768061 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:59.768066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:59.768069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:59.768263 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:59.768596 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:59.768601 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:59.768839 (+ 238us) client_negotiation.cc:769] Sending connection context
0925 02:55:59.769017 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 02:55:59.769297 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":150,"threads_started":1}
I20260925 02:55:59.771265 29597 heartbeater.cc:347] Connected to a master server at 127.12.222.254:42831
I20260925 02:55:59.771471 29597 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:59.771864 29597 heartbeater.cc:511] Master 127.12.222.254:42831 requested a full tablet report, sending...
I20260925 02:55:59.772861 29266 ts_manager.cc:194] Registered new tserver with Master: b62ca49e40f943a5b442416394149281 (127.12.222.194:43343)
I20260925 02:55:59.773556 29266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:55765
W20260925 02:55:59.861855 29600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:55:59.862063 29600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:55:59.862115 29600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:55:59.865561 29600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:55:59.865610 29600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:55:59.865682 29600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:55:59.870401 29600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:36371
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=45237
--tserver_master_addrs=127.12.222.254:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:55:59.871172 29600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:55:59.871992 29600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:55:59.880458 29608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:59.880472 29605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:55:59.880473 29606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:55:59.880568 29600 server_base.cc:1034] running on GCE node
I20260925 02:55:59.881016 29600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:55:59.881578 29600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:55:59.882766 29600 hybrid_clock.cc:648] HybridClock initialized: now 1790304959882749 us; error 27 us; skew 500 ppm
Sep 25 02:55:59 dist-test-slave-252r krb5kdc[28682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304959, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:55:59.885277 29600 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:55:59.886125 29600 webserver.cc:460] Webserver started at http://127.12.222.195:45237/ using document root <none> and password file <none>
I20260925 02:55:59.886647 29600 fs_manager.cc:359] Metadata directory not provided
I20260925 02:55:59.886744 29600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:55:59.890136 29600 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:55:59.891954 29615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:59.893159 29600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:55:59.893291 29600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "747fcb3d278946acb50e1b79daf99ca9"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "875cd534352d9c71a8c4d6d2ebe01b7c"
server_key_iv: "1a6fa3d1ac6d220109dfde4f8ac39b3c"
server_key_version: "encryptionkey@0"
I20260925 02:55:59.893754 29600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:55:59.909092 29600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:55:59.912364 29600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:55:59.912535 29600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:55:59.913045 29600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:55:59.913766 29600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:55:59.913872 29600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:59.913934 29600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:55:59.914000 29600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:55:59.923599 29600 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:36371
I20260925 02:55:59.923621 29729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:36371 every 8 connection(s)
I20260925 02:55:59.924548 29600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:55:59 dist-test-slave-252r krb5kdc[28682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304959, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:55:59.934512 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 29600
I20260925 02:55:59.937223 29459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:59.926147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:42041 (local address 127.12.222.254:42831)
0925 02:55:59.926275 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:59.926279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:59.926969 (+ 690us) server_negotiation.cc:407] Connection header received
0925 02:55:59.928122 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:59.928126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:59.928182 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:59.928336 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:59.929441 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.930028 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.931197 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.931407 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.931480 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:59.933691 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:59.933694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:59.933696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:59.933716 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:59.935276 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:59.935769 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:59.935772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:59.935773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:59.935863 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:59.936179 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:59.936183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:59.936185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:59.936403 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:59.936509 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:59.936961 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 02:55:59.937073 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:55:59.937906 29732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:59.926459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:42041)
0925 02:55:59.926840 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:59.926876 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:59.927928 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:59.928528 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:59.928534 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:59.928840 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:55:59.929245 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:59.929257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.930136 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.930139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:59.931095 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:59.931105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.931520 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.931522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:59.931640 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:59.932140 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:55:59.932154 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:55:59.933537 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:55:59.935384 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:59.935389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:59.935407 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:59.935673 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:59.935953 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:55:59.935956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:55:59.935957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:55:59.936090 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:55:59.936522 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:55:59.936530 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:55:59.936736 (+ 206us) client_negotiation.cc:769] Sending connection context
0925 02:55:59.936921 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 02:55:59.937186 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":112,"threads_started":1}
I20260925 02:55:59.938971 29730 heartbeater.cc:347] Connected to a master server at 127.12.222.254:42831
I20260925 02:55:59.939190 29730 heartbeater.cc:464] Registering TS with master...
I20260925 02:55:59.939615 29730 heartbeater.cc:511] Master 127.12.222.254:42831 requested a full tablet report, sending...
I20260925 02:55:59.940682 29266 ts_manager.cc:194] Registered new tserver with Master: 747fcb3d278946acb50e1b79daf99ca9 (127.12.222.195:36371)
I20260925 02:55:59.941417 29266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:42041
I20260925 02:55:59.942939 29733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:59.935156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54256 (local address 127.12.222.254:42831)
0925 02:55:59.935358 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:59.935363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:59.935594 (+ 231us) server_negotiation.cc:407] Connection header received
0925 02:55:59.935667 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:59.935670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:59.935717 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:59.935798 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:55:59.936532 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.937008 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.937941 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.938229 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.938326 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:59.939649 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:55:59.939653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:55:59.939655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:55:59.939678 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:55:59.941429 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:59.941908 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:59.941910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:59.941911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:59.942012 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:55:59.942262 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:55:59.942265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:55:59.942266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:55:59.942455 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:55:59.942643 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:59.942715 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 02:55:59.942814 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260925 02:55:59.944476 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:55:59.947925 29733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:59.945025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54258 (local address 127.12.222.254:42831)
0925 02:55:59.945128 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:59.945132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:59.945233 (+ 101us) server_negotiation.cc:407] Connection header received
0925 02:55:59.945359 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:59.945361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:59.945443 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:59.945510 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:55:59.945669 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.946136 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.946883 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.947075 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.947145 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:59.947377 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:55:59.947644 (+ 267us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:55:59.947717 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:59.947774 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 02:55:59.947843 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:55:59.948464 29266 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:55:59.951426 29266 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:55:59.958504 29389 tablet_service.cc:1467] Processing CreateTablet for tablet 2edd2266fe0b46de87d867dd60e84ab2 (DEFAULT_TABLE table=test-table [id=d3139c09821f4296bbec08c04850bc05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:59.959108 29389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2edd2266fe0b46de87d867dd60e84ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:59.961882 29347 tablet_bootstrap.cc:492] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd: Bootstrap starting.
I20260925 02:55:59.962426 29464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:59.957350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:43343 (local address 127.0.0.1:44024)
0925 02:55:59.957525 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:59.957538 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:59.957603 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:59.958498 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:59.958500 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:59.958515 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:59.958574 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:59.958578 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.959782 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.959785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:59.960942 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:59.960972 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.962010 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.962014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:59.962123 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:59.962167 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 02:55:59.962216 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 02:55:59.962298 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":3}
I20260925 02:55:59.962697 29347 tablet_bootstrap.cc:654] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:59.963002 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:59.957597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.12.222.194:43343)
0925 02:55:59.958065 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:59.958077 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:59.958097 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:55:59.958168 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:59.958172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:59.958361 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:59.958509 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:59.958683 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.959650 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.961124 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.961889 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.962657 (+ 768us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:59.962782 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:59.962830 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 02:55:59.962909 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":132,"threads_started":1}
I20260925 02:55:59.963939 29347 tablet_bootstrap.cc:492] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd: No bootstrap required, opened a new log
I20260925 02:55:59.964027 29347 ts_tablet_manager.cc:1397] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:55:59.964483 29347 raft_consensus.cc:348] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:59.964598 29347 raft_consensus.cc:374] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:59.964641 29347 raft_consensus.cc:729] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e949ce4de3d147de854bbe61e96dc8fd, State: Initialized, Role: FOLLOWER
I20260925 02:55:59.964715 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:59.958030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43934 (local address 127.12.222.195:36371)
0925 02:55:59.958250 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:59.958260 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:59.960454 (+ 2194us) server_negotiation.cc:407] Connection header received
0925 02:55:59.960570 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:59.960575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:59.960722 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:59.960906 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:59.961114 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.961715 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.963431 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.964154 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.964291 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:59.964403 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:59.964494 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 02:55:59.964576 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260925 02:55:59.964752 29347 consensus_queue.cc:260] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:59.964926 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:59.957837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:36371 (local address 127.0.0.1:43934)
0925 02:55:59.960364 (+ 2527us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:59.960375 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:59.960471 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:59.960882 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:59.960885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:59.960902 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:59.961016 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:59.961023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.961869 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.961872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:59.963317 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:59.963327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.964275 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.964279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:59.964366 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:59.964387 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 02:55:59.964465 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:55:59.964524 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2477,"mutex_wait_us":1,"thread_start_us":64,"threads_started":1}
I20260925 02:55:59.965015 29347 ts_tablet_manager.cc:1428] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:55:59.965793 29530 tablet_service.cc:1467] Processing CreateTablet for tablet 2edd2266fe0b46de87d867dd60e84ab2 (DEFAULT_TABLE table=test-table [id=d3139c09821f4296bbec08c04850bc05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:59.966593 29530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2edd2266fe0b46de87d867dd60e84ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:59.966981 29664 tablet_service.cc:1467] Processing CreateTablet for tablet 2edd2266fe0b46de87d867dd60e84ab2 (DEFAULT_TABLE table=test-table [id=d3139c09821f4296bbec08c04850bc05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:55:59.967628 29664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2edd2266fe0b46de87d867dd60e84ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:55:59.972209 29743 tablet_bootstrap.cc:492] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281: Bootstrap starting.
I20260925 02:55:59.972976 29744 tablet_bootstrap.cc:492] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: Bootstrap starting.
I20260925 02:55:59.974818 29743 tablet_bootstrap.cc:654] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:59.974841 29744 tablet_bootstrap.cc:654] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: Neither blocks nor log segments found. Creating new log.
I20260925 02:55:59.975440 29744 log.cc:824] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:59.975451 29743 log.cc:824] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281: Log is configured to *not* fsync() on all Append() calls
I20260925 02:55:59.977128 29743 tablet_bootstrap.cc:492] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281: No bootstrap required, opened a new log
I20260925 02:55:59.977140 29744 tablet_bootstrap.cc:492] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: No bootstrap required, opened a new log
I20260925 02:55:59.977293 29743 ts_tablet_manager.cc:1397] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 02:55:59.977311 29744 ts_tablet_manager.cc:1397] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 02:55:59.979673 29743 raft_consensus.cc:348] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:59.979673 29744 raft_consensus.cc:348] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:59.979863 29743 raft_consensus.cc:374] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:59.979889 29744 raft_consensus.cc:374] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:55:59.979919 29743 raft_consensus.cc:729] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b62ca49e40f943a5b442416394149281, State: Initialized, Role: FOLLOWER
I20260925 02:55:59.979955 29744 raft_consensus.cc:729] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747fcb3d278946acb50e1b79daf99ca9, State: Initialized, Role: FOLLOWER
I20260925 02:55:59.980345 29743 consensus_queue.cc:260] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:59.980381 29744 consensus_queue.cc:260] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:59.981009 29597 heartbeater.cc:503] Master 127.12.222.254:42831 was elected leader, sending a full tablet report...
I20260925 02:55:59.981302 29744 ts_tablet_manager.cc:1428] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 02:55:59.981300 29743 ts_tablet_manager.cc:1428] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 02:55:59.981611 29730 heartbeater.cc:503] Master 127.12.222.254:42831 was elected leader, sending a full tablet report...
I20260925 02:55:59.990463 29463 raft_consensus.cc:482] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:55:59.990581 29463 raft_consensus.cc:504] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:59.991436 29463 leader_election.cc:288] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b62ca49e40f943a5b442416394149281 (127.12.222.194:43343), 747fcb3d278946acb50e1b79daf99ca9 (127.12.222.195:36371)
I20260925 02:55:59.995581 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:59.991738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:58721 (local address 127.12.222.195:36371)
0925 02:55:59.991850 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:59.991854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:59.992053 (+ 199us) server_negotiation.cc:407] Connection header received
0925 02:55:59.992142 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:59.992147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:59.992187 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:59.992276 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:59.992517 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.993137 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.994413 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.994994 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.995105 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:59.995145 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:59.995425 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 02:55:59.995481 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:55:59.995581 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:59.991617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:60881 (local address 127.12.222.194:43343)
0925 02:55:59.991714 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 02:55:59.991718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:55:59.991782 (+ 64us) server_negotiation.cc:407] Connection header received
0925 02:55:59.991958 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:55:59.991960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:55:59.991993 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:55:59.992082 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:55:59.992367 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.993028 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.994417 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.994994 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.995105 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:59.995145 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 02:55:59.995426 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 02:55:59.995481 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:55:59.995637 29460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:59.991549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:43343 (local address 127.12.222.193:60881)
0925 02:55:59.991703 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:59.991717 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:59.991850 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:59.992103 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:59.992108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:59.992125 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:59.992266 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:59.992276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.993126 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.993128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:59.994295 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:59.994305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.995107 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.995110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:59.995274 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:59.995324 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 02:55:59.995398 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:55:59.995503 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":56}
I20260925 02:55:59.995731 29749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:55:59.991673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:36371 (local address 127.12.222.193:58721)
0925 02:55:59.991976 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 02:55:59.991986 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:55:59.992071 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:55:59.992298 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:55:59.992303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:55:59.992331 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:55:59.992431 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:59.992435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.993223 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.993226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:59.994298 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:55:59.994305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:55:59.995107 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:55:59.995110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:55:59.995262 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:55:59.995327 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 02:55:59.995398 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 02:55:59.995503 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":97,"thread_start_us":116,"threads_started":1}
I20260925 02:55:59.996166 29550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edd2266fe0b46de87d867dd60e84ab2" candidate_uuid: "e949ce4de3d147de854bbe61e96dc8fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b62ca49e40f943a5b442416394149281" is_pre_election: true
I20260925 02:55:59.996206 29684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edd2266fe0b46de87d867dd60e84ab2" candidate_uuid: "e949ce4de3d147de854bbe61e96dc8fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747fcb3d278946acb50e1b79daf99ca9" is_pre_election: true
I20260925 02:55:59.996433 29550 raft_consensus.cc:2454] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e949ce4de3d147de854bbe61e96dc8fd in term 0.
I20260925 02:55:59.996479 29684 raft_consensus.cc:2454] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e949ce4de3d147de854bbe61e96dc8fd in term 0.
I20260925 02:55:59.996893 29343 leader_election.cc:302] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b62ca49e40f943a5b442416394149281, e949ce4de3d147de854bbe61e96dc8fd; no voters:
I20260925 02:55:59.997166 29463 raft_consensus.cc:2781] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:55:59.997259 29463 raft_consensus.cc:482] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:55:59.997318 29463 raft_consensus.cc:3037] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:59.998266 29463 raft_consensus.cc:504] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:55:59.998533 29463 leader_election.cc:288] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [CANDIDATE]: Term 1 election: Requested vote from peers b62ca49e40f943a5b442416394149281 (127.12.222.194:43343), 747fcb3d278946acb50e1b79daf99ca9 (127.12.222.195:36371)
I20260925 02:55:59.998790 29550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edd2266fe0b46de87d867dd60e84ab2" candidate_uuid: "e949ce4de3d147de854bbe61e96dc8fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b62ca49e40f943a5b442416394149281"
I20260925 02:55:59.998898 29550 raft_consensus.cc:3037] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:59.998904 29684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edd2266fe0b46de87d867dd60e84ab2" candidate_uuid: "e949ce4de3d147de854bbe61e96dc8fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747fcb3d278946acb50e1b79daf99ca9"
I20260925 02:55:59.999040 29684 raft_consensus.cc:3037] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:55:59.999861 29550 raft_consensus.cc:2454] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e949ce4de3d147de854bbe61e96dc8fd in term 1.
I20260925 02:56:00.000006 29684 raft_consensus.cc:2454] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e949ce4de3d147de854bbe61e96dc8fd in term 1.
I20260925 02:56:00.000164 29343 leader_election.cc:302] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b62ca49e40f943a5b442416394149281, e949ce4de3d147de854bbe61e96dc8fd; no voters:
I20260925 02:56:00.000397 29463 raft_consensus.cc:2781] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:00.000557 29463 raft_consensus.cc:686] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 1 LEADER]: Becoming Leader. State: Replica: e949ce4de3d147de854bbe61e96dc8fd, State: Running, Role: LEADER
I20260925 02:56:00.000799 29463 consensus_queue.cc:237] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:56:00.003328 29265 catalog_manager.cc:5495] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd reported cstate change: term changed from 0 to 1, leader changed from <none> to e949ce4de3d147de854bbe61e96dc8fd (127.12.222.193). New cstate: current_term: 1 leader_uuid: "e949ce4de3d147de854bbe61e96dc8fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } health_report { overall_health: HEALTHY } } }
W20260925 02:56:00.007385 29598 tablet.cc:2369] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:00.019142 29465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:00.016160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36904 (local address 127.12.222.193:44251)
0925 02:56:00.016285 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:00.016290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:00.016355 (+ 65us) server_negotiation.cc:407] Connection header received
0925 02:56:00.016518 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:00.016525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:00.016568 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:00.016675 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:00.016909 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:00.017374 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:00.018115 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:00.018334 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:00.018431 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:00.018674 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:00.018790 (+ 116us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 02:56:00.019012 (+ 222us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36904: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 02:56:00.019210 29465 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36904: token expired
I20260925 02:56:00.027357 29733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:00.019882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54264 (local address 127.12.222.254:42831)
0925 02:56:00.019988 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:00.019992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:00.020012 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:56:00.020041 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:00.020043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:00.020074 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:00.020134 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:00.020812 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:00.021395 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:00.022541 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:00.022747 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:00.022866 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:00.024174 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:00.024178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:00.024180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:00.024204 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:00.025765 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:00.026185 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:00.026188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:00.026190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:00.026268 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:00.026550 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:00.026555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:00.026556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:00.026717 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:00.026868 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:00.027112 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 02:56:00.027221 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:56:00.031426 29465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:00.028637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.12.222.193:44251)
0925 02:56:00.028711 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:00.028715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:00.028743 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:56:00.028847 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:00.028849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:00.028880 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:00.028926 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:00.029131 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:00.029666 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:00.030466 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:00.030654 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:00.030725 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:00.030913 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:00.031180 (+ 267us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:00.031236 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:00.031311 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 02:56:00.031350 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:56:00.034560 29550 raft_consensus.cc:1261] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281 [term 1 FOLLOWER]: Refusing update from remote peer e949ce4de3d147de854bbe61e96dc8fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:00.034581 29684 raft_consensus.cc:1261] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9 [term 1 FOLLOWER]: Refusing update from remote peer e949ce4de3d147de854bbe61e96dc8fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:00.035179 29463 consensus_queue.cc:1035] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:00.035358 29750 consensus_queue.cc:1035] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:00.043362 29754 mvcc.cc:205] Tried to move back new op lower bound from 7333089116296499200 to 7333089116165439488. Current Snapshot: MvccSnapshot[applied={T|T < 7333089116296499200}]
I20260925 02:56:00.043699 29755 mvcc.cc:205] Tried to move back new op lower bound from 7333089116296499200 to 7333089116165439488. Current Snapshot: MvccSnapshot[applied={T|T < 7333089116296499200}]
I20260925 02:56:00.043999 29753 mvcc.cc:205] Tried to move back new op lower bound from 7333089116296499200 to 7333089116165439488. Current Snapshot: MvccSnapshot[applied={T|T < 7333089116296499200 or (T in {7333089116296499200})}]
I20260925 02:56:00.050235 29264 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:00.050392 29264 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:00.052580 29264 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: Sending DeleteTablet for 3 replicas of tablet 2edd2266fe0b46de87d867dd60e84ab2
I20260925 02:56:00.053179 29530 tablet_service.cc:1514] Processing DeleteTablet for tablet 2edd2266fe0b46de87d867dd60e84ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:00 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:44024
I20260925 02:56:00.053216 29664 tablet_service.cc:1514] Processing DeleteTablet for tablet 2edd2266fe0b46de87d867dd60e84ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:00 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:43934
I20260925 02:56:00.053301 29389 tablet_service.cc:1514] Processing DeleteTablet for tablet 2edd2266fe0b46de87d867dd60e84ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:00 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:36900
I20260925 02:56:00.053435 29466 tablet_replica.cc:330] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd: stopping tablet replica
I20260925 02:56:00.053547 29466 raft_consensus.cc:2229] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:56:00.053678 29764 tablet_replica.cc:330] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: stopping tablet replica
I20260925 02:56:00.053819 29466 raft_consensus.cc:2258] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:00.053942 29764 raft_consensus.cc:2229] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:00.054204 29764 raft_consensus.cc:2258] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:00.054244 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 29305
I20260925 02:56:00.056488 29764 ts_tablet_manager.cc:1905] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:00.058594 29764 ts_tablet_manager.cc:1918] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:56:00.058668 29764 log.cc:1196] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/2edd2266fe0b46de87d867dd60e84ab2
I20260925 02:56:00.058916 29763 tablet_replica.cc:330] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281: stopping tablet replica
I20260925 02:56:00.059046 29764 ts_tablet_manager.cc:1939] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: Deleting consensus metadata
I20260925 02:56:00.059212 29763 raft_consensus.cc:2229] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:00.059477 29763 raft_consensus.cc:2258] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:00.060186 29251 catalog_manager.cc:4841] TS 747fcb3d278946acb50e1b79daf99ca9 (127.12.222.195:36371): tablet 2edd2266fe0b46de87d867dd60e84ab2 (table test-table [id=d3139c09821f4296bbec08c04850bc05]) successfully deleted
I20260925 02:56:00.060801 29763 ts_tablet_manager.cc:1905] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 02:56:00.061151 29250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.222.193:44251 (error 108)
I20260925 02:56:00.061945 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 29461
I20260925 02:56:00.062145 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:00.061831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:44251 (local address 127.0.0.1:36926)
0925 02:56:00.061967 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:00.062045 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.193:44251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 02:56:00.062551 29250 catalog_manager.cc:4584] TS e949ce4de3d147de854bbe61e96dc8fd (127.12.222.193:44251): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2edd2266fe0b46de87d867dd60e84ab2: Network error: Client connection negotiation failed: client connection to 127.12.222.193:44251: connect: Connection refused (error 111)
I20260925 02:56:00.067701 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 29600
I20260925 02:56:00.068773 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:00.067805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:43343 (local address 127.0.0.1:44038)
0925 02:56:00.068635 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:00.068693 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.194:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":748}
I20260925 02:56:00.073325 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 29233
I20260925 02:56:20.079866 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:42831
--webserver_interface=127.12.222.254
--webserver_port=33493
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:42831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:20.179636 29766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:20.179841 29766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:20.179876 29766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:20.183578 29766 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 02:56:20.183619 29766 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 02:56:20.183632 29766 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 02:56:20.183645 29766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:20.183660 29766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:20.183676 29766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:20.183689 29766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:20.183702 29766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:20.188406 29766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:42831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:42831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=33493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:20.189261 29766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:20.190141 29766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:20.198510 29774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:20.198529 29766 server_base.cc:1034] running on GCE node
W20260925 02:56:20.198565 29772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:20.198511 29771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:20.199420 29766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:20.200434 29766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:20.201581 29766 hybrid_clock.cc:648] HybridClock initialized: now 1790304980201555 us; error 38 us; skew 500 ppm
Sep 25 02:56:20 dist-test-slave-252r krb5kdc[28682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304980, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:20.203975 29766 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:56:20.204840 29766 webserver.cc:460] Webserver started at http://127.12.222.254:33493/ using document root <none> and password file <none>
I20260925 02:56:20.205335 29766 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:20.205425 29766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:20.208889 29766 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:56:20.210764 29782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:20.211895 29766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:20.212034 29766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "1120e5447cea410abdc1c24c44670e3e"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "20f48fb3e5838c1dcd50d4c47d3bd80d"
server_key_iv: "1b2be8ea11bf8392cfade5406c47f104"
server_key_version: "encryptionkey@0"
I20260925 02:56:20.212579 29766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:20.232584 29766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:20.235044 29766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:20.235219 29766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:20.241909 29766 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:42831
I20260925 02:56:20.242184 29834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:42831 every 8 connection(s)
I20260925 02:56:20.242909 29766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:20.246536 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 29766
I20260925 02:56:20.246877 29835 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:56:20.246980 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:44251
--local_ip_for_outbound_sockets=127.12.222.193
--tserver_master_addrs=127.12.222.254:42831
--webserver_port=43407
--webserver_interface=127.12.222.193
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:56:20.248440 29835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: Bootstrap starting.
I20260925 02:56:20.264456 29835 log.cc:824] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:20.274549 29835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:56:20.274897 29835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: Bootstrap complete.
I20260925 02:56:20.278630 29835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1120e5447cea410abdc1c24c44670e3e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42831 } }
I20260925 02:56:20.279019 29835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1120e5447cea410abdc1c24c44670e3e, State: Initialized, Role: FOLLOWER
I20260925 02:56:20.279501 29835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1120e5447cea410abdc1c24c44670e3e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42831 } }
I20260925 02:56:20.279641 29835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:20.279739 29835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:20.279848 29835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:56:20.281605 29835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1120e5447cea410abdc1c24c44670e3e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42831 } }
I20260925 02:56:20.281910 29835 leader_election.cc:302] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1120e5447cea410abdc1c24c44670e3e; no voters:
I20260925 02:56:20.282212 29835 leader_election.cc:288] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 02:56:20.282290 29839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 3 FOLLOWER]: Leader election won for term 3
I20260925 02:56:20.282513 29839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [term 3 LEADER]: Becoming Leader. State: Replica: 1120e5447cea410abdc1c24c44670e3e, State: Running, Role: LEADER
I20260925 02:56:20.282776 29839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1120e5447cea410abdc1c24c44670e3e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42831 } }
I20260925 02:56:20.283290 29835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:20.283564 29840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1120e5447cea410abdc1c24c44670e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1120e5447cea410abdc1c24c44670e3e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42831 } } }
I20260925 02:56:20.283582 29839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1120e5447cea410abdc1c24c44670e3e. Latest consensus state: current_term: 3 leader_uuid: "1120e5447cea410abdc1c24c44670e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1120e5447cea410abdc1c24c44670e3e" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42831 } } }
I20260925 02:56:20.283762 29839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:20.283829 29840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:20.284116 29843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:20.288712 29843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:20.289330 29843 catalog_manager.cc:1259] Loaded cluster ID: 71d2b48eafde4c72b4e48ea6b200e18b
I20260925 02:56:20.289390 29843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:20.291788 29843 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:20.292526 29843 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: Loaded TSK: 0
I20260925 02:56:20.292606 29843 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1120e5447cea410abdc1c24c44670e3e: Loaded TSK: 1
F20260925 02:56:20.293323 29843 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f68d43e79ad google::LogMessage::Fail() at ??:0
@ 0x7f68d43ea88a google::LogMessage::SendToLog() at ??:0
@ 0x7f68d43e74de google::LogMessage::Flush() at ??:0
@ 0x7f68d43e8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f68d70ab18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f68d70a3f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f68ded4c7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f68ded2699c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f68ded27178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f68ded68df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f68ded8782b std::function<>::operator()() at ??:0
@ 0x7f68ded26b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f68ded27f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f68ded24579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f68ded68640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f68d66393d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f68d6639c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f68d663b3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f68d662a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f68d7c6e6db start_thread at ??:0
@ 0x7f68d2ee561f clone at ??:0
W20260925 02:56:20.384454 29837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:20.384742 29837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:20.384802 29837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:20.390723 29837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:20.390785 29837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:20.390861 29837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:56:20.398027 29837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:44251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=43407
--tserver_master_addrs=127.12.222.254:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:20.399251 29837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:20.400496 29837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:20.410431 29889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:20.410444 29890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:20.410694 29892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:20.411275 29837 server_base.cc:1034] running on GCE node
I20260925 02:56:20.411695 29837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:20.412412 29837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:20.413610 29837 hybrid_clock.cc:648] HybridClock initialized: now 1790304980413578 us; error 39 us; skew 500 ppm
Sep 25 02:56:20 dist-test-slave-252r krb5kdc[28682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304980, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:20.416635 29837 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:56:20.417652 29837 webserver.cc:460] Webserver started at http://127.12.222.193:43407/ using document root <none> and password file <none>
I20260925 02:56:20.418267 29837 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:20.418354 29837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:20.423095 29837 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:56:20.425442 29899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:20.426874 29837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:56:20.426995 29837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "e949ce4de3d147de854bbe61e96dc8fd"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "6eca6ee7824474566447bfe162b81ddd"
server_key_iv: "c18348562244faa6b8e0555ab7a33846"
server_key_version: "encryptionkey@0"
I20260925 02:56:20.427649 29837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:20.444216 29837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:20.447389 29837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:20.447546 29837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:20.448088 29837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:20.449257 29906 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 02:56:20.452821 29837 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:56:20.452886 29837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260925 02:56:20.452940 29837 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:56:20.454799 29837 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:56:20.454854 29837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 02:56:20.455114 29906 tablet_bootstrap.cc:492] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd: Bootstrap starting.
I20260925 02:56:20.465065 29837 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:44251
I20260925 02:56:20.465389 30015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:44251 every 8 connection(s)
I20260925 02:56:20.466100 29837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:56:20.473168 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 29837
I20260925 02:56:20.473650 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:43343
--local_ip_for_outbound_sockets=127.12.222.194
--tserver_master_addrs=127.12.222.254:42831
--webserver_port=35971
--webserver_interface=127.12.222.194
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:56:20.473728 29906 log.cc:824] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:20.479871 29906 tablet_bootstrap.cc:492] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:56:20.480187 29906 tablet_bootstrap.cc:492] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd: Bootstrap complete.
I20260925 02:56:20.481235 29906 ts_tablet_manager.cc:1397] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd: Time spent bootstrapping tablet: real 0.026s user 0.021s sys 0.004s
I20260925 02:56:20.483338 29906 raft_consensus.cc:348] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:56:20.483717 29906 raft_consensus.cc:729] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e949ce4de3d147de854bbe61e96dc8fd, State: Initialized, Role: FOLLOWER
I20260925 02:56:20.484136 29906 consensus_queue.cc:260] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:56:20.484946 29906 ts_tablet_manager.cc:1428] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260925 02:56:20.539489 30018 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:56:20.468156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:57117)
0925 02:56:20.468718 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:20.468771 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:20.469872 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:20.538755 (+ 68883us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":161,"threads_started":1}
W20260925 02:56:20.539935 30016 heartbeater.cc:644] Failed to heartbeat to 127.12.222.254:42831 (0 consecutive failures): Network error: Failed to ping master at 127.12.222.254:42831: Client connection negotiation failed: client connection to 127.12.222.254:42831: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:56:20.540657 30018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:20.540405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:56583)
0925 02:56:20.540523 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:20.540605 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 02:56:20.541348 30018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:20.541116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:42839)
0925 02:56:20.541235 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:20.541287 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 02:56:20.541534 30016 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:56:20.594153 30020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:20.594336 30020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:20.594372 30020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:20.598035 30020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:20.598078 30020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:20.598126 30020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:56:20.602792 30020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:43343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=35971
--tserver_master_addrs=127.12.222.254:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:20.603648 30020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:20.604507 30020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:20.612363 30027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:20.612408 30026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:20.612569 30020 server_base.cc:1034] running on GCE node
W20260925 02:56:20.612644 30029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:20.612974 30020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:20.613492 30020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:20.614665 30020 hybrid_clock.cc:648] HybridClock initialized: now 1790304980614656 us; error 31 us; skew 500 ppm
Sep 25 02:56:20 dist-test-slave-252r krb5kdc[28682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304980, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:20.617069 30020 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:56:20.617826 30020 webserver.cc:460] Webserver started at http://127.12.222.194:35971/ using document root <none> and password file <none>
I20260925 02:56:20.618287 30020 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:20.618342 30020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:20.621699 30020 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:56:20.623622 30036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:20.624698 30020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:56:20.624782 30020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "b62ca49e40f943a5b442416394149281"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "2a85cee3a76ff2d41208ac988d7aa54a"
server_key_iv: "802bd88d473cd62dc63687ca9ee1d025"
server_key_version: "encryptionkey@0"
I20260925 02:56:20.625182 30020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:20.639405 30020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:20.641860 30020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:20.642009 30020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:20.642433 30020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:20.643317 30043 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 02:56:20.646636 30020 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:56:20.646687 30020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260925 02:56:20.646727 30020 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:56:20.648062 30020 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:56:20.648102 30020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 02:56:20.648406 30043 tablet_bootstrap.cc:492] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281: Bootstrap starting.
I20260925 02:56:20.653770 30043 log.cc:824] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:20.657745 30020 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:43343
I20260925 02:56:20.657807 30153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:43343 every 8 connection(s)
I20260925 02:56:20.658519 30020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:56:20.661182 30043 tablet_bootstrap.cc:492] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:56:20.661454 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:20.660252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:56985)
0925 02:56:20.660727 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:20.660833 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":155,"threads_started":1}
I20260925 02:56:20.661571 30043 tablet_bootstrap.cc:492] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281: Bootstrap complete.
W20260925 02:56:20.661864 30159 heartbeater.cc:644] Failed to heartbeat to 127.12.222.254:42831 (0 consecutive failures): Network error: Failed to ping master at 127.12.222.254:42831: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
I20260925 02:56:20.661927 30043 ts_tablet_manager.cc:1397] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260925 02:56:20.662436 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:20.662234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:45105)
0925 02:56:20.662332 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:20.662374 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 02:56:20.663101 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:20.662919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:47209)
0925 02:56:20.663006 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:20.663044 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 02:56:20.663312 30159 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:56:20.664208 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 30020
I20260925 02:56:20.664681 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:36371
--local_ip_for_outbound_sockets=127.12.222.195
--tserver_master_addrs=127.12.222.254:42831
--webserver_port=45237
--webserver_interface=127.12.222.195
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:56:20.665176 30043 raft_consensus.cc:348] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:56:20.665776 30043 raft_consensus.cc:729] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b62ca49e40f943a5b442416394149281, State: Initialized, Role: FOLLOWER
I20260925 02:56:20.666224 30043 consensus_queue.cc:260] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:56:20.667155 30043 ts_tablet_manager.cc:1428] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 02:56:20.667241 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:20.666978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:35195)
0925 02:56:20.667095 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:20.667154 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 02:56:20.667865 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:20.667630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:38987)
0925 02:56:20.667750 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:20.667801 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 02:56:20.765197 30162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:20.765406 30162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:20.765442 30162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:20.768875 30162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:20.768916 30162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:20.768965 30162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:56:20.773574 30162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:36371
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=45237
--tserver_master_addrs=127.12.222.254:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:20.774271 30162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:20.775058 30162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:20.783762 30169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:20.783743 30171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:20.783752 30168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:20.784137 30162 server_base.cc:1034] running on GCE node
I20260925 02:56:20.784514 30162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:20.785064 30162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:20.786216 30162 hybrid_clock.cc:648] HybridClock initialized: now 1790304980786189 us; error 38 us; skew 500 ppm
Sep 25 02:56:20 dist-test-slave-252r krb5kdc[28682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304980, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:20.788655 30162 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:56:20.789335 30162 webserver.cc:460] Webserver started at http://127.12.222.195:45237/ using document root <none> and password file <none>
I20260925 02:56:20.789784 30162 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:20.789841 30162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:20.793267 30162 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260925 02:56:20.795023 30178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:20.796146 30162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:20.796291 30162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "747fcb3d278946acb50e1b79daf99ca9"
format_stamp: "Formatted at 2026-09-25 02:55:38 on dist-test-slave-252r"
server_key: "875cd534352d9c71a8c4d6d2ebe01b7c"
server_key_iv: "1a6fa3d1ac6d220109dfde4f8ac39b3c"
server_key_version: "encryptionkey@0"
I20260925 02:56:20.796778 30162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:20.831413 30162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:20.834270 30162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:20.834453 30162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:20.835002 30162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:20.835762 30162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:20.835826 30162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:20.835891 30162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:20.835943 30162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:20.844621 30162 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:36371
I20260925 02:56:20.844656 30292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:36371 every 8 connection(s)
I20260925 02:56:20.845504 30162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:56:20.849216 30295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:20.847533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:43725)
0925 02:56:20.847948 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:20.848048 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":120,"threads_started":1}
W20260925 02:56:20.849661 30293 heartbeater.cc:644] Failed to heartbeat to 127.12.222.254:42831 (0 consecutive failures): Network error: Failed to ping master at 127.12.222.254:42831: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
I20260925 02:56:20.850227 30295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:20.850020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:48567)
0925 02:56:20.850123 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:20.850171 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 02:56:20.850786 30295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:20.850591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:44191)
0925 02:56:20.850689 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:20.850733 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 02:56:20.850960 30293 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:56:20.851742 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 30162
I20260925 02:56:21.542413 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:21.542026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:59181)
0925 02:56:21.542256 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:21.542305 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":65,"threads_started":1}
I20260925 02:56:21.668968 30298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:21.668561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:46375)
0925 02:56:21.668811 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:21.668870 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":59,"threads_started":1}
I20260925 02:56:21.675155 30299 raft_consensus.cc:482] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:56:21.675339 30299 raft_consensus.cc:504] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:56:21.676424 30299 leader_election.cc:288] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b62ca49e40f943a5b442416394149281 (127.12.222.194:43343), 747fcb3d278946acb50e1b79daf99ca9 (127.12.222.195:36371)
Sep 25 02:56:21 dist-test-slave-252r krb5kdc[28682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304980, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.194@KRBTEST.COM
Sep 25 02:56:21 dist-test-slave-252r krb5kdc[28682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304980, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.195@KRBTEST.COM
I20260925 02:56:21.689255 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:21.676631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:43343 (local address 127.12.222.193:48519)
0925 02:56:21.677342 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:21.677353 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:21.677426 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:21.678458 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:21.678464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:21.678999 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:21.679450 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:21.679464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:21.680695 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:21.680700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:21.681612 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:21.681621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:21.682307 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:21.682311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:21.682533 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:21.683167 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:21.683182 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:21.684780 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:21.687024 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:21.687029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:21.687040 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:21.687295 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:21.687639 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:21.687642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:21.687644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:21.687775 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:21.688383 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:21.688389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:21.688590 (+ 201us) client_negotiation.cc:769] Sending connection context
0925 02:56:21.688759 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 02:56:21.689049 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":185,"spinlock_wait_cycles":1280,"thread_start_us":76,"threads_started":1}
I20260925 02:56:21.689268 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:21.676803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:48519 (local address 127.12.222.194:43343)
0925 02:56:21.677591 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:21.677600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:21.677626 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:56:21.677818 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:21.677822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:21.678128 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:21.678350 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:21.679609 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:21.680566 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:21.681748 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:21.682170 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:21.682333 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:21.684965 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:21.684970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:21.684980 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:21.685003 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:21.686900 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:21.687421 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:21.687431 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:21.687434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:21.687540 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:21.687872 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:21.687877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:21.687879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:21.688223 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:21.688403 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:21.688860 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 02:56:21.689083 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":74,"threads_started":1}
I20260925 02:56:21.689298 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:21.676631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:36371 (local address 127.12.222.193:57161)
0925 02:56:21.676966 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:21.676983 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:21.677082 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:21.678597 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:21.678601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:21.678987 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:21.679450 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:21.679464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:21.680722 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:21.680724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:21.681609 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:21.681618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:21.682279 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:21.682283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:21.682532 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:21.683211 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:21.683223 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:21.685565 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:21.687406 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:21.687410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:21.687413 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:21.687598 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:21.687933 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:21.687936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:21.687938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:21.688036 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:21.688615 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:21.688617 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:21.688700 (+ 83us) client_negotiation.cc:769] Sending connection context
0925 02:56:21.688810 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 02:56:21.689133 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":234,"spinlock_wait_cycles":8192}
I20260925 02:56:21.689606 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:21.677105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:57161 (local address 127.12.222.195:36371)
0925 02:56:21.677368 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:21.677380 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:21.677421 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:56:21.678017 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:21.678022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:21.678342 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:21.678624 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:21.679613 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:21.680584 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:21.681725 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:21.682161 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:21.682316 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:21.685776 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:21.685781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:21.685790 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:21.685813 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:21.687291 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:21.687710 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:21.687715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:21.687721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:21.687830 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:21.688121 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:21.688125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:21.688127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:21.688513 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:21.688642 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:21.688940 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 02:56:21.689200 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":79,"threads_started":1}
I20260925 02:56:21.690435 30106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edd2266fe0b46de87d867dd60e84ab2" candidate_uuid: "e949ce4de3d147de854bbe61e96dc8fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b62ca49e40f943a5b442416394149281" is_pre_election: true
I20260925 02:56:21.690703 30106 raft_consensus.cc:2454] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e949ce4de3d147de854bbe61e96dc8fd in term 1.
I20260925 02:56:21.691635 29902 leader_election.cc:302] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b62ca49e40f943a5b442416394149281, e949ce4de3d147de854bbe61e96dc8fd; no voters:
W20260925 02:56:21.691854 29901 leader_election.cc:341] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 747fcb3d278946acb50e1b79daf99ca9 (127.12.222.195:36371): Not found: Tablet not found: 2edd2266fe0b46de87d867dd60e84ab2
I20260925 02:56:21.691990 30299 raft_consensus.cc:2781] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 02:56:21.692061 30299 raft_consensus.cc:482] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:56:21.692107 30299 raft_consensus.cc:3037] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:56:21.694059 30299 raft_consensus.cc:504] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:56:21.694353 30299 leader_election.cc:288] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [CANDIDATE]: Term 2 election: Requested vote from peers b62ca49e40f943a5b442416394149281 (127.12.222.194:43343), 747fcb3d278946acb50e1b79daf99ca9 (127.12.222.195:36371)
I20260925 02:56:21.694698 30106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edd2266fe0b46de87d867dd60e84ab2" candidate_uuid: "e949ce4de3d147de854bbe61e96dc8fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b62ca49e40f943a5b442416394149281"
I20260925 02:56:21.694828 30106 raft_consensus.cc:3037] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281 [term 1 FOLLOWER]: Advancing to term 2
W20260925 02:56:21.694983 29901 leader_election.cc:341] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 747fcb3d278946acb50e1b79daf99ca9 (127.12.222.195:36371): Not found: Tablet not found: 2edd2266fe0b46de87d867dd60e84ab2
I20260925 02:56:21.696897 30106 raft_consensus.cc:2454] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e949ce4de3d147de854bbe61e96dc8fd in term 2.
I20260925 02:56:21.697180 29902 leader_election.cc:302] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b62ca49e40f943a5b442416394149281, e949ce4de3d147de854bbe61e96dc8fd; no voters: 747fcb3d278946acb50e1b79daf99ca9
I20260925 02:56:21.697335 30299 raft_consensus.cc:2781] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:56:21.697497 30299 raft_consensus.cc:686] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [term 2 LEADER]: Becoming Leader. State: Replica: e949ce4de3d147de854bbe61e96dc8fd, State: Running, Role: LEADER
I20260925 02:56:21.697817 30299 consensus_queue.cc:237] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:56:21.698120 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:21.697911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:36941)
0925 02:56:21.698013 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:21.698060 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 02:56:21.851861 30305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:21.851409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:44637)
0925 02:56:21.851730 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:21.851780 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":74,"threads_started":1}
I20260925 02:56:22.098796 30106 raft_consensus.cc:1261] T 2edd2266fe0b46de87d867dd60e84ab2 P b62ca49e40f943a5b442416394149281 [term 2 FOLLOWER]: Refusing update from remote peer e949ce4de3d147de854bbe61e96dc8fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 02:56:22.099488 30299 consensus_queue.cc:1035] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 02:56:22.101496 30298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:22.101213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:43563)
0925 02:56:22.101371 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:22.101426 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 02:56:22.103943 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:22.103711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:37345)
0925 02:56:22.103821 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:22.103872 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 02:56:22.103994 30298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:22.103789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:48763)
0925 02:56:22.103894 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:22.103935 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 02:56:22.104312 29901 consensus_peers.cc:487] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd -> Peer 747fcb3d278946acb50e1b79daf99ca9 (127.12.222.195:36371): Couldn't send request to peer 747fcb3d278946acb50e1b79daf99ca9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2edd2266fe0b46de87d867dd60e84ab2. This is attempt 1: this message will repeat every 5th retry.
I20260925 02:56:22.694566 30311 ts_tablet_manager.cc:927] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: Initiating tablet copy from peer e949ce4de3d147de854bbe61e96dc8fd (127.12.222.193:44251)
I20260925 02:56:22.695164 30311 tablet_copy_client.cc:316] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: tablet copy: Beginning tablet copy session from remote peer at address 127.12.222.193:44251
Sep 25 02:56:22 dist-test-slave-252r krb5kdc[28682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304980, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.193@KRBTEST.COM
I20260925 02:56:22.704114 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:22.695583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:44251 (local address 127.12.222.195:55685)
0925 02:56:22.695815 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:22.695834 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:22.695948 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:22.696412 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:22.696417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:22.696788 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:22.696894 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:22.696904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:22.697792 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:22.697795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:22.698516 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:22.698525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:22.699094 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:22.699096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:22.699212 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:22.699699 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:22.699710 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:22.701111 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:22.702794 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:22.702802 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:22.702806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:22.703019 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:22.703248 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:22.703250 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:22.703251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:22.703316 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:22.703698 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:22.703703 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:22.703800 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 02:56:22.703901 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 02:56:22.704007 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":97,"threads_started":1}
I20260925 02:56:22.704222 30314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:22.695733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:55685 (local address 127.12.222.193:44251)
0925 02:56:22.695962 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:22.695971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:22.696001 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:56:22.696045 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:22.696049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:22.696219 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:22.696400 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:22.697004 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:22.697674 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:22.698639 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:22.698993 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:22.699077 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:22.701242 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:22.701252 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:22.701262 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:22.701285 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:22.702688 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:22.703104 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:22.703108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:22.703109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:22.703178 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:22.703397 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:22.703399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:22.703401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:22.703611 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:22.703703 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:22.703974 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 02:56:22.704077 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20260925 02:56:22.705050 29990 tablet_copy_service.cc:140] P e949ce4de3d147de854bbe61e96dc8fd: Received BeginTabletCopySession request for tablet 2edd2266fe0b46de87d867dd60e84ab2 from peer 747fcb3d278946acb50e1b79daf99ca9 ({username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:55685)
I20260925 02:56:22.705209 29990 tablet_copy_service.cc:161] P e949ce4de3d147de854bbe61e96dc8fd: Beginning new tablet copy session on tablet 2edd2266fe0b46de87d867dd60e84ab2 from peer 747fcb3d278946acb50e1b79daf99ca9 at {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:55685: session id = 747fcb3d278946acb50e1b79daf99ca9-2edd2266fe0b46de87d867dd60e84ab2
I20260925 02:56:22.706203 29990 tablet_copy_source_session.cc:205] T 2edd2266fe0b46de87d867dd60e84ab2 P e949ce4de3d147de854bbe61e96dc8fd: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 02:56:22.709962 30311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2edd2266fe0b46de87d867dd60e84ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:22.715759 30311 tablet_copy_client.cc:791] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: tablet copy: Starting download of 0 data blocks...
I20260925 02:56:22.715982 30311 tablet_copy_client.cc:655] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: tablet copy: Starting download of 1 WAL segments...
I20260925 02:56:22.717505 30311 tablet_copy_client.cc:531] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 02:56:22.718858 30311 tablet_bootstrap.cc:492] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: Bootstrap starting.
I20260925 02:56:22.723690 30311 log.cc:824] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:22.729387 30311 tablet_bootstrap.cc:492] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:56:22.729674 30311 tablet_bootstrap.cc:492] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: Bootstrap complete.
I20260925 02:56:22.729993 30311 ts_tablet_manager.cc:1397] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260925 02:56:22.732026 30311 raft_consensus.cc:348] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:56:22.732281 30311 raft_consensus.cc:729] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747fcb3d278946acb50e1b79daf99ca9, State: Initialized, Role: FOLLOWER
I20260925 02:56:22.732656 30311 consensus_queue.cc:260] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62ca49e40f943a5b442416394149281" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 43343 } } peers { permanent_uuid: "747fcb3d278946acb50e1b79daf99ca9" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 36371 } } peers { permanent_uuid: "e949ce4de3d147de854bbe61e96dc8fd" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 44251 } }
I20260925 02:56:22.733526 30311 ts_tablet_manager.cc:1428] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20260925 02:56:22.733608 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:22.733368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:42135)
0925 02:56:22.733471 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:22.733519 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 02:56:22.734051 29990 tablet_copy_service.cc:342] P e949ce4de3d147de854bbe61e96dc8fd: Request end of tablet copy session 747fcb3d278946acb50e1b79daf99ca9-2edd2266fe0b46de87d867dd60e84ab2 received from {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:55685
I20260925 02:56:22.734143 29990 tablet_copy_service.cc:433] P e949ce4de3d147de854bbe61e96dc8fd: ending tablet copy session 747fcb3d278946acb50e1b79daf99ca9-2edd2266fe0b46de87d867dd60e84ab2 on tablet 2edd2266fe0b46de87d867dd60e84ab2 with peer 747fcb3d278946acb50e1b79daf99ca9
I20260925 02:56:22.734236 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:22.734026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:44529)
0925 02:56:22.734118 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:22.734170 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 02:56:23.105034 30317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:23.104663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:42365)
0925 02:56:23.104888 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:23.104943 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":80,"threads_started":1}
I20260925 02:56:23.105135 30318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:23.104663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:39301)
0925 02:56:23.104967 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:23.105014 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":82,"threads_started":1}
I20260925 02:56:23.316190 30318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:23.315925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:38791)
0925 02:56:23.316060 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:23.316114 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 02:56:23.316368 30247 raft_consensus.cc:1203] T 2edd2266fe0b46de87d867dd60e84ab2 P 747fcb3d278946acb50e1b79daf99ca9 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260925 02:56:23.317663 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:23.317336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:50251)
0925 02:56:23.317550 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:23.317596 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":83,"threads_started":1}
I20260925 02:56:24.106226 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:24.105805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:39013)
0925 02:56:24.106076 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:24.106137 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":71,"threads_started":1}
I20260925 02:56:24.317301 30324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:24.316851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:44019)
0925 02:56:24.317158 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:24.317209 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":69,"threads_started":1}
I20260925 02:56:24.318595 30325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:24.318243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:48481)
0925 02:56:24.318456 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:24.318522 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260925 02:56:25.107380 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:25.106999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:41487)
0925 02:56:25.107223 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:25.107272 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
I20260925 02:56:25.318434 30327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:25.318073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:33761)
0925 02:56:25.318288 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:25.318336 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":66,"threads_started":1}
I20260925 02:56:25.319578 30328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:25.319191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:59361)
0925 02:56:25.319450 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:25.319502 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
I20260925 02:56:26.108659 30329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:26.108307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:56303)
0925 02:56:26.108523 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:26.108569 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":72,"threads_started":1}
I20260925 02:56:26.319692 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:26.319281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:54067)
0925 02:56:26.319539 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:26.319591 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
I20260925 02:56:26.320456 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:26.320181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:38385)
0925 02:56:26.320348 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:26.320399 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 02:56:26.320839 30332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:26.320418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:59991)
0925 02:56:26.320704 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:26.320755 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":76,"threads_started":1}
I20260925 02:56:27.109668 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:27.109293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:52407)
0925 02:56:27.109531 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:27.109577 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
I20260925 02:56:27.321640 30334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:27.321201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:45741)
0925 02:56:27.321483 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:27.321547 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":110,"threads_started":1}
I20260925 02:56:27.321913 30335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:27.321538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:37825)
0925 02:56:27.321752 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:27.321813 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":76,"threads_started":1}
I20260925 02:56:28.110796 30336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:28.110419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:42077)
0925 02:56:28.110646 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:28.110697 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260925 02:56:28.322789 30337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:28.322390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:53667)
0925 02:56:28.322642 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:28.322695 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
I20260925 02:56:28.323020 30338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:28.322569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:60479)
0925 02:56:28.322822 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:28.322873 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":106,"threads_started":1}
I20260925 02:56:29.112130 30339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:29.111625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:44401)
0925 02:56:29.111983 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:29.112036 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":63,"threads_started":1}
I20260925 02:56:29.324002 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:29.323545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:50817)
0925 02:56:29.323803 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:29.323865 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260925 02:56:29.324159 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:29.323737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:54289)
0925 02:56:29.323980 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:29.324033 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260925 02:56:30.113179 30342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:30.112800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:58811)
0925 02:56:30.113028 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:30.113089 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260925 02:56:30.325273 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:30.324837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:34813)
0925 02:56:30.325135 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:30.325184 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":85,"threads_started":1}
I20260925 02:56:30.325342 30344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:30.324970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:54525)
0925 02:56:30.325215 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:30.325263 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":57,"threads_started":1}
I20260925 02:56:31.114355 30345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:31.113947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:57715)
0925 02:56:31.114206 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:31.114266 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":94,"threads_started":1}
I20260925 02:56:31.326563 30346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:31.326172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:37305)
0925 02:56:31.326425 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:31.326475 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":82,"threads_started":1}
I20260925 02:56:31.326596 30347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:31.326180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:41369)
0925 02:56:31.326475 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:31.326521 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":81,"threads_started":1}
I20260925 02:56:32.115535 30348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:32.115087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:49733)
0925 02:56:32.115384 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:32.115447 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":64,"threads_started":1}
I20260925 02:56:32.327752 30349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:32.327345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:35397)
0925 02:56:32.327593 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:32.327651 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":67,"threads_started":1}
I20260925 02:56:32.327939 30350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:32.327484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:59781)
0925 02:56:32.327747 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:32.327806 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260925 02:56:33.116832 30351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:33.116438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:36289)
0925 02:56:33.116693 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:33.116742 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":86,"threads_started":1}
I20260925 02:56:33.328774 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:33.328394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:50279)
0925 02:56:33.328636 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:33.328686 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":91,"threads_started":1}
I20260925 02:56:33.328979 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:33.328545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:42781)
0925 02:56:33.328803 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:33.328863 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":106,"threads_started":1}
I20260925 02:56:34.117862 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:34.117480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:59073)
0925 02:56:34.117701 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:34.117767 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
I20260925 02:56:34.329742 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:34.329356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:52959)
0925 02:56:34.329607 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:34.329658 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":96,"threads_started":1}
I20260925 02:56:34.329969 30357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:34.329611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:54341)
0925 02:56:34.329805 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:34.329864 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260925 02:56:35.118906 30358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.118468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.194:58373)
0925 02:56:35.118770 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:35.118819 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":65,"threads_started":1}
I20260925 02:56:35.330857 30359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.330468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.195:35009)
0925 02:56:35.330719 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:35.330771 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":91,"threads_started":1}
I20260925 02:56:35.330952 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:35.330587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42831 (local address 127.12.222.193:50039)
0925 02:56:35.330801 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:35.330847 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.254:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":67,"threads_started":1}
I20260925 02:56:35.860292 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 29837
I20260925 02:56:35.866415 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 30020
I20260925 02:56:35.871549 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 30162
2026-09-25T02:56:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58755 ms)
[----------] 30 tests from SecurityITest (232945 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:35 dist-test-slave-252r krb5kdc[30363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:35 dist-test-slave-252r krb5kdc[30363](info): set up 2 sockets
Sep 25 02:56:35 dist-test-slave-252r krb5kdc[30363](info): commencing operation
krb5kdc: starting...
W20260925 02:56:36.847757 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:56:36 dist-test-slave-252r krb5kdc[30363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:56:36.955327 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:38031
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:36217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:38031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:37.054294 30376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:37.054508 30376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:37.054558 30376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:37.058056 30376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:37.058109 30376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:37.058125 30376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:37.058158 30376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:37.058171 30376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:37.062916 30376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:36217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:38031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:38031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:37.063753 30376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:37.064632 30376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:37.072482 30381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:37.072526 30384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:37.072489 30382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:37.072664 30376 server_base.cc:1034] running on GCE node
I20260925 02:56:37.073202 30376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:37.074196 30376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:37.075362 30376 hybrid_clock.cc:648] HybridClock initialized: now 1790304997075344 us; error 39 us; skew 500 ppm
Sep 25 02:56:37 dist-test-slave-252r krb5kdc[30363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304997, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:37.077756 30376 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:56:37.078447 30376 webserver.cc:460] Webserver started at http://127.12.222.254:42395/ using document root <none> and password file <none>
I20260925 02:56:37.078914 30376 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:37.078967 30376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:37.079106 30376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:37.080948 30376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "f330fa3ca6084e9991de752dfb314ded"
format_stamp: "Formatted at 2026-09-25 02:56:37 on dist-test-slave-252r"
server_key: "3771e28fd41310f04e464e8740fe5069"
server_key_iv: "af24fe3cb75ec3facd55f41f54ed9cac"
server_key_version: "encryptionkey@0"
I20260925 02:56:37.081408 30376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "f330fa3ca6084e9991de752dfb314ded"
format_stamp: "Formatted at 2026-09-25 02:56:37 on dist-test-slave-252r"
server_key: "3771e28fd41310f04e464e8740fe5069"
server_key_iv: "af24fe3cb75ec3facd55f41f54ed9cac"
server_key_version: "encryptionkey@0"
I20260925 02:56:37.084540 30376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:56:37.086598 30391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:37.087594 30376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:37.087706 30376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "f330fa3ca6084e9991de752dfb314ded"
format_stamp: "Formatted at 2026-09-25 02:56:37 on dist-test-slave-252r"
server_key: "3771e28fd41310f04e464e8740fe5069"
server_key_iv: "af24fe3cb75ec3facd55f41f54ed9cac"
server_key_version: "encryptionkey@0"
I20260925 02:56:37.087795 30376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:37.099581 30376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:37.102186 30376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:37.102362 30376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:37.108717 30376 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:38031
I20260925 02:56:37.108737 30443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:38031 every 8 connection(s)
I20260925 02:56:37.109508 30376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:37.110033 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 30376
I20260925 02:56:37.110172 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:56:37.110435 13179 external_mini_cluster.cc:1428] Setting key 1d5bc8a5fe393ada646c64ad6ad47a43
I20260925 02:56:37.112434 30444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:37.118430 30444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded: Bootstrap starting.
Sep 25 02:56:37 dist-test-slave-252r krb5kdc[30363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:56:37.120836 30444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:37.121657 30444 log.cc:824] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:37.123448 30444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded: No bootstrap required, opened a new log
I20260925 02:56:37.124022 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:37.111734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.12.222.254:38031)
0925 02:56:37.112120 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:37.112137 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:37.112177 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:56:37.112919 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:37.112939 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:37.113321 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:37.113666 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:37.114325 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.115688 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.116527 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.117131 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.117307 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:37.119505 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:37.119512 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:37.119526 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:37.119555 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:37.121405 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:37.121840 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:37.121844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:37.121848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:37.121967 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:37.122263 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:37.122269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:37.122271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:37.122646 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:37.122837 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:37.123122 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 02:56:37.123363 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":107,"threads_started":1}
I20260925 02:56:37.127020 30444 raft_consensus.cc:348] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f330fa3ca6084e9991de752dfb314ded" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38031 } }
I20260925 02:56:37.127243 30444 raft_consensus.cc:374] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:37.127300 30444 raft_consensus.cc:729] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f330fa3ca6084e9991de752dfb314ded, State: Initialized, Role: FOLLOWER
I20260925 02:56:37.127687 30444 consensus_queue.cc:260] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f330fa3ca6084e9991de752dfb314ded" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38031 } }
I20260925 02:56:37.127835 30444 raft_consensus.cc:388] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:37.127909 30444 raft_consensus.cc:482] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:37.127993 30444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:37.128760 30444 raft_consensus.cc:504] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f330fa3ca6084e9991de752dfb314ded" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38031 } }
I20260925 02:56:37.129024 30444 leader_election.cc:302] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f330fa3ca6084e9991de752dfb314ded; no voters:
I20260925 02:56:37.129235 30444 leader_election.cc:288] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:37.129359 30449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:37.129614 30449 raft_consensus.cc:686] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded [term 1 LEADER]: Becoming Leader. State: Replica: f330fa3ca6084e9991de752dfb314ded, State: Running, Role: LEADER
I20260925 02:56:37.129912 30449 consensus_queue.cc:237] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f330fa3ca6084e9991de752dfb314ded" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38031 } }
I20260925 02:56:37.130286 30444 sys_catalog.cc:564] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:37.131520 30451 sys_catalog.cc:455] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded [sys.catalog]: SysCatalogTable state changed. Reason: New leader f330fa3ca6084e9991de752dfb314ded. Latest consensus state: current_term: 1 leader_uuid: "f330fa3ca6084e9991de752dfb314ded" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f330fa3ca6084e9991de752dfb314ded" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38031 } } }
I20260925 02:56:37.131547 30450 sys_catalog.cc:455] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f330fa3ca6084e9991de752dfb314ded" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f330fa3ca6084e9991de752dfb314ded" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38031 } } }
I20260925 02:56:37.131639 30451 sys_catalog.cc:458] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:37.131646 30450 sys_catalog.cc:458] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:37.132045 30458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:37.134903 30458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:37.139825 30458 catalog_manager.cc:1347] Generated new cluster ID: d4136e12b1174becb04f610d71fd30e8
I20260925 02:56:37.139886 30458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:37.164705 30458 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:37.165455 30458 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:37.175565 30458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded: Generated new TSK 0
I20260925 02:56:37.176151 30458 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
I20260925 02:56:37.207175 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38031
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:36217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:37.322132 30470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:37.322333 30470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:37.322398 30470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:37.326120 30470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:37.326162 30470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:37.326208 30470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:56:37.330756 30470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:36217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:37.331554 30470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:37.332511 30470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:37.340874 30475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:37.340894 30478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:37.340967 30470 server_base.cc:1034] running on GCE node
W20260925 02:56:37.340869 30476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:37.341544 30470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:37.342120 30470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:37.343279 30470 hybrid_clock.cc:648] HybridClock initialized: now 1790304997343267 us; error 23 us; skew 500 ppm
Sep 25 02:56:37 dist-test-slave-252r krb5kdc[30363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304997, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:37.345775 30470 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:56:37.346592 30470 webserver.cc:460] Webserver started at http://127.12.222.193:32991/ using document root <none> and password file <none>
I20260925 02:56:37.347082 30470 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:37.347136 30470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:37.347361 30470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:37.349135 30470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "5fda004723794b4ea19f807f56c8ba61"
format_stamp: "Formatted at 2026-09-25 02:56:37 on dist-test-slave-252r"
server_key: "35978a143172ea5c4f6eabc15d85975d"
server_key_iv: "2a234b6d07387f10fd6830004297c24e"
server_key_version: "encryptionkey@0"
I20260925 02:56:37.349643 30470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "5fda004723794b4ea19f807f56c8ba61"
format_stamp: "Formatted at 2026-09-25 02:56:37 on dist-test-slave-252r"
server_key: "35978a143172ea5c4f6eabc15d85975d"
server_key_iv: "2a234b6d07387f10fd6830004297c24e"
server_key_version: "encryptionkey@0"
I20260925 02:56:37.353264 30470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:56:37.355353 30485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:37.356429 30470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:56:37.356542 30470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "5fda004723794b4ea19f807f56c8ba61"
format_stamp: "Formatted at 2026-09-25 02:56:37 on dist-test-slave-252r"
server_key: "35978a143172ea5c4f6eabc15d85975d"
server_key_iv: "2a234b6d07387f10fd6830004297c24e"
server_key_version: "encryptionkey@0"
I20260925 02:56:37.356638 30470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:37.375110 30470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:37.377486 30470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:37.377667 30470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:37.378237 30470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:37.379071 30470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:37.379132 30470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:37.379176 30470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:37.379204 30470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:37.387750 30470 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:40009
I20260925 02:56:37.387774 30598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:40009 every 8 connection(s)
I20260925 02:56:37.388582 30470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:56:37.392685 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 30470
I20260925 02:56:37.392799 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:56:37.393091 13179 external_mini_cluster.cc:1428] Setting key 1fbda03e1b58c076654481eb77afbd77
Sep 25 02:56:37 dist-test-slave-252r krb5kdc[30363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304997, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:56:37.401768 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:37.390143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:49289 (local address 127.12.222.254:38031)
0925 02:56:37.390309 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:37.390314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:37.390996 (+ 682us) server_negotiation.cc:407] Connection header received
0925 02:56:37.392263 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:37.392267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:37.392320 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:37.392422 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:37.393594 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.394222 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.395339 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.395716 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.395868 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:37.398230 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:37.398236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:37.398242 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:37.398273 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:37.399738 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:37.400366 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:37.400369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:37.400370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:37.400440 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:37.400809 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:37.400811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:37.400812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:37.400968 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:37.401036 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:37.401577 (+ 541us) server_negotiation.cc:299] Negotiation successful
0925 02:56:37.401666 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:56:37.402787 30601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:37.390450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38031 (local address 127.12.222.193:49289)
0925 02:56:37.390863 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:37.390904 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:37.392056 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:37.392575 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:37.392581 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:37.392932 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:37.393410 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:37.393429 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.394358 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.394361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:37.395217 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:37.395226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.395895 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.395901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:37.396122 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:37.396624 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:37.396644 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:37.398081 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:37.399860 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:37.399868 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:37.399882 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:37.400262 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:37.400530 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:37.400535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:37.400538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:37.400710 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:37.401074 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:37.401080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:37.401345 (+ 265us) client_negotiation.cc:769] Sending connection context
0925 02:56:37.401538 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 02:56:37.401840 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.12.222.194@KRBTEST.COM; defaulting to no policy
I20260925 02:56:37.403905 30599 heartbeater.cc:347] Connected to a master server at 127.12.222.254:38031
I20260925 02:56:37.404083 30599 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:37.404587 30599 heartbeater.cc:511] Master 127.12.222.254:38031 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.194@KRBTEST.COM" created.
I20260925 02:56:37.405866 30408 ts_manager.cc:194] Registered new tserver with Master: 5fda004723794b4ea19f807f56c8ba61 (127.12.222.193:40009)
I20260925 02:56:37.406878 30408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:49289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
I20260925 02:56:37.420441 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38031
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:36217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:37.523283 30604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:37.523518 30604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:37.523592 30604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:37.527238 30604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:37.527316 30604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:37.527400 30604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:56:37.532368 30604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:36217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:37.533233 30604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:37.534125 30604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:37.543010 30610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:37.543081 30609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:37.543180 30604 server_base.cc:1034] running on GCE node
W20260925 02:56:37.543081 30612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:37.543685 30604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:37.544219 30604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:37.545431 30604 hybrid_clock.cc:648] HybridClock initialized: now 1790304997545412 us; error 29 us; skew 500 ppm
Sep 25 02:56:37 dist-test-slave-252r krb5kdc[30363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304997, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:37.547890 30604 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:56:37.548753 30604 webserver.cc:460] Webserver started at http://127.12.222.194:35429/ using document root <none> and password file <none>
I20260925 02:56:37.549263 30604 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:37.549342 30604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:37.549566 30604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:37.551364 30604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "a95f1703bc1848fba1ee7274f74ffba9"
format_stamp: "Formatted at 2026-09-25 02:56:37 on dist-test-slave-252r"
server_key: "b2174a25fdb48b65fc74758e0c7508e5"
server_key_iv: "99c59d28582efee1d792b31242763e7a"
server_key_version: "encryptionkey@0"
I20260925 02:56:37.551913 30604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "a95f1703bc1848fba1ee7274f74ffba9"
format_stamp: "Formatted at 2026-09-25 02:56:37 on dist-test-slave-252r"
server_key: "b2174a25fdb48b65fc74758e0c7508e5"
server_key_iv: "99c59d28582efee1d792b31242763e7a"
server_key_version: "encryptionkey@0"
I20260925 02:56:37.555227 30604 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:56:37.557294 30619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:37.558382 30604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:56:37.558504 30604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "a95f1703bc1848fba1ee7274f74ffba9"
format_stamp: "Formatted at 2026-09-25 02:56:37 on dist-test-slave-252r"
server_key: "b2174a25fdb48b65fc74758e0c7508e5"
server_key_iv: "99c59d28582efee1d792b31242763e7a"
server_key_version: "encryptionkey@0"
I20260925 02:56:37.558607 30604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:37.585217 30604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:37.587620 30604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:37.587792 30604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:37.588359 30604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:37.589073 30604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:37.589139 30604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:37.589205 30604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:37.589246 30604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:37.597357 30604 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:34699
I20260925 02:56:37.597385 30732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:34699 every 8 connection(s)
I20260925 02:56:37.598073 30604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:56:37.605888 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 30604
I20260925 02:56:37.606050 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:56:37.606351 13179 external_mini_cluster.cc:1428] Setting key 983d600fd79ea14fd65e5fa4265f22cf
Sep 25 02:56:37 dist-test-slave-252r krb5kdc[30363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304997, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:56:37.610898 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:37.599604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:45251 (local address 127.12.222.254:38031)
0925 02:56:37.599726 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:37.599730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:37.600466 (+ 736us) server_negotiation.cc:407] Connection header received
0925 02:56:37.601750 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:37.601754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:37.601824 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:37.601922 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:37.602917 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.603469 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.604633 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.604890 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.605002 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:37.607360 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:37.607363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:37.607366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:37.607400 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:37.608790 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:37.609382 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:37.609385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:37.609386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:37.609456 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:37.609787 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:37.609791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:37.609794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:37.610033 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:37.610136 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:37.610550 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 02:56:37.610673 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:56:37.611348 30735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:37.599907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38031 (local address 127.12.222.194:45251)
0925 02:56:37.600337 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:37.600373 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:37.601543 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:37.602100 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:37.602106 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:37.602387 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:37.602760 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:37.602771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.603598 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.603602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:37.604520 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:37.604529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.605040 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.605045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:37.605197 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:37.605652 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:37.605667 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:37.607208 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:37.608908 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:37.608916 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:37.608931 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:37.609275 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:37.609549 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:37.609553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:37.609557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:37.609688 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:37.610134 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:37.610138 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:37.610332 (+ 194us) client_negotiation.cc:769] Sending connection context
0925 02:56:37.610464 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 02:56:37.610658 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":99,"threads_started":1}
I20260925 02:56:37.612511 30733 heartbeater.cc:347] Connected to a master server at 127.12.222.254:38031
I20260925 02:56:37.612712 30733 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:37.613129 30733 heartbeater.cc:511] Master 127.12.222.254:38031 requested a full tablet report, sending...
I20260925 02:56:37.614171 30408 ts_manager.cc:194] Registered new tserver with Master: a95f1703bc1848fba1ee7274f74ffba9 (127.12.222.194:34699)
I20260925 02:56:37.614940 30408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:45251
WARNING: no policy specified for kudu/127.12.222.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
I20260925 02:56:37.632323 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38031
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:36217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:37.735643 30738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:37.735914 30738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:37.735997 30738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:37.739899 30738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:37.739969 30738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:37.740060 30738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:56:37.745103 30738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:36217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:38031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:37.745996 30738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:37.746865 30738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:37.755730 30744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:37.755784 30743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:37.755754 30746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:37.755836 30738 server_base.cc:1034] running on GCE node
I20260925 02:56:37.756428 30738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:37.756985 30738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:37.758163 30738 hybrid_clock.cc:648] HybridClock initialized: now 1790304997758151 us; error 31 us; skew 500 ppm
Sep 25 02:56:37 dist-test-slave-252r krb5kdc[30363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304997, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:37.760979 30738 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:56:37.761652 30738 webserver.cc:460] Webserver started at http://127.12.222.195:36043/ using document root <none> and password file <none>
I20260925 02:56:37.762221 30738 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:37.762279 30738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:37.762418 30738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:37.764210 30738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "c51fb9333e714a44967490d4bbb96595"
format_stamp: "Formatted at 2026-09-25 02:56:37 on dist-test-slave-252r"
server_key: "f2841406cb3031c66b9bfb6f825e11b3"
server_key_iv: "4a1922fa8b3a4ed97a840e337bd8e6cf"
server_key_version: "encryptionkey@0"
I20260925 02:56:37.764744 30738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "c51fb9333e714a44967490d4bbb96595"
format_stamp: "Formatted at 2026-09-25 02:56:37 on dist-test-slave-252r"
server_key: "f2841406cb3031c66b9bfb6f825e11b3"
server_key_iv: "4a1922fa8b3a4ed97a840e337bd8e6cf"
server_key_version: "encryptionkey@0"
I20260925 02:56:37.767971 30738 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:56:37.769970 30753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:37.770982 30738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:37.771087 30738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "c51fb9333e714a44967490d4bbb96595"
format_stamp: "Formatted at 2026-09-25 02:56:37 on dist-test-slave-252r"
server_key: "f2841406cb3031c66b9bfb6f825e11b3"
server_key_iv: "4a1922fa8b3a4ed97a840e337bd8e6cf"
server_key_version: "encryptionkey@0"
I20260925 02:56:37.771178 30738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:37.781276 30738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:37.783712 30738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:37.783901 30738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:37.784485 30738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:37.785235 30738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:37.785279 30738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:37.785341 30738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:37.785382 30738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:37.793776 30738 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:46511
I20260925 02:56:37.793793 30866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:46511 every 8 connection(s)
I20260925 02:56:37.794553 30738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:56:37.797495 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 30738
I20260925 02:56:37.797646 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:56:37.798045 13179 external_mini_cluster.cc:1428] Setting key d8ae3e2ce11a1bec41b1d145a8743b99
Sep 25 02:56:37 dist-test-slave-252r krb5kdc[30363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304997, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:56:37.807116 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:37.796333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:34467 (local address 127.12.222.254:38031)
0925 02:56:37.796457 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:37.796460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:37.797185 (+ 725us) server_negotiation.cc:407] Connection header received
0925 02:56:37.798729 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:37.798731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:37.798796 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:37.798872 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:37.800328 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.800870 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.801892 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.802102 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.802168 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:37.804318 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:37.804321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:37.804323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:37.804344 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:37.805541 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:37.805979 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:37.805982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:37.805983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:37.806048 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:37.806338 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:37.806340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:37.806341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:37.806493 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:37.806571 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:37.806933 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 02:56:37.807013 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:56:37.807767 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:37.796657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38031 (local address 127.12.222.195:34467)
0925 02:56:37.797058 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:37.797095 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:37.798517 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:37.799167 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:37.799176 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:37.799605 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:37.800176 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:37.800196 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.800977 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.800980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:37.801789 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:37.801798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.802212 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.802215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:37.802334 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:37.802780 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:37.802793 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:37.804129 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:37.805647 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:37.805651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:37.805660 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:37.805897 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:37.806134 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:37.806135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:37.806137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:37.806264 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:37.806584 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:37.806588 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:37.806760 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 02:56:37.806890 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 02:56:37.807082 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":109,"threads_started":1}
I20260925 02:56:37.808763 30867 heartbeater.cc:347] Connected to a master server at 127.12.222.254:38031
I20260925 02:56:37.808956 30867 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:37.809357 30867 heartbeater.cc:511] Master 127.12.222.254:38031 requested a full tablet report, sending...
I20260925 02:56:37.810191 30408 ts_manager.cc:194] Registered new tserver with Master: c51fb9333e714a44967490d4bbb96595 (127.12.222.195:46511)
I20260925 02:56:37.810698 30408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:34467
I20260925 02:56:37.811769 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:56:37.819551 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:37.813581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58972 (local address 127.12.222.254:38031)
0925 02:56:37.813679 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:37.813682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:37.813892 (+ 210us) server_negotiation.cc:407] Connection header received
0925 02:56:37.813984 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:37.813986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:37.814016 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:37.814079 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:37.814673 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.815088 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.815824 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.816025 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.816090 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:37.817205 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:37.817207 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:37.817209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:37.817227 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:37.818256 (+ 1029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:37.818619 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:37.818622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:37.818623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:37.818687 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:37.818907 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:37.818909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:37.818910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:37.819100 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:37.819195 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:37.819379 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 02:56:37.819456 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:56:37.821978 30408 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:56:37.824721 30408 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:56:37.837550 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:37.832592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51432 (local address 127.12.222.193:40009)
0925 02:56:37.833005 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:37.833021 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:37.833045 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:56:37.833124 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:37.833129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:37.833324 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:37.833820 (+ 496us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:37.833972 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.834748 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.836264 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.836935 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.837054 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:37.837161 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:37.837323 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 02:56:37.837407 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":109,"threads_started":1}
I20260925 02:56:37.837548 30880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:37.831909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:40009 (local address 127.0.0.1:51432)
0925 02:56:37.832762 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:37.832811 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:37.832955 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:37.833523 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:37.833529 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:37.833549 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:37.833677 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:37.833684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.834874 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.834879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:37.836114 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:37.836124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.837047 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.837050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:37.837147 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:37.837181 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 02:56:37.837296 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 02:56:37.837404 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":106,"spinlock_wait_cycles":23168,"thread_start_us":85,"threads_started":1}
I20260925 02:56:37.837584 30882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:37.831791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:34699 (local address 127.0.0.1:54532)
0925 02:56:37.832762 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:37.832813 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:37.832957 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:37.833659 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:37.833661 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:37.833680 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:37.833757 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:37.833762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.834923 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.834926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:37.836081 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:37.836091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.837192 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.837195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:37.837277 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:37.837300 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:56:37.837369 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 02:56:37.837430 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":50,"spinlock_wait_cycles":768,"thread_start_us":66,"threads_started":1}
I20260925 02:56:37.838073 30884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:37.832801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.12.222.195:46511)
0925 02:56:37.833225 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:37.833240 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:37.833450 (+ 210us) server_negotiation.cc:407] Connection header received
0925 02:56:37.833652 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:37.833657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:37.833815 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:37.833927 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:37.834302 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.835255 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.836611 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.837321 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.837423 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:37.837534 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:37.837836 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 02:56:37.837937 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":129,"threads_started":1}
I20260925 02:56:37.838078 30881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:37.831915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.12.222.194:34699)
0925 02:56:37.832556 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:37.832568 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:37.833109 (+ 541us) server_negotiation.cc:407] Connection header received
0925 02:56:37.833193 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:37.833199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:37.833388 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:37.833543 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:37.833862 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.834769 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.836201 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.836928 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.837029 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:37.837131 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:37.837841 (+ 710us) server_negotiation.cc:299] Negotiation successful
0925 02:56:37.837946 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":137,"threads_started":1}
I20260925 02:56:37.838420 30879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:37.832230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:46511 (local address 127.0.0.1:53936)
0925 02:56:37.833364 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:37.833382 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:37.833522 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:37.834074 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:37.834076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:37.834092 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:37.834174 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:37.834180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.835388 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.835398 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:37.836492 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:37.836502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:37.837591 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:37.837597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:37.837680 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:37.837703 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:56:37.838235 (+ 532us) client_negotiation.cc:240] Negotiation successful
0925 02:56:37.838305 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
I20260925 02:56:37.840720 30801 tablet_service.cc:1467] Processing CreateTablet for tablet 4637b961328f435888f41f987fc9aefb (DEFAULT_TABLE table=test-table [id=6c2547635e9943eda418ed85e0c0d7e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:37.840721 30533 tablet_service.cc:1467] Processing CreateTablet for tablet 4637b961328f435888f41f987fc9aefb (DEFAULT_TABLE table=test-table [id=6c2547635e9943eda418ed85e0c0d7e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:37.840721 30667 tablet_service.cc:1467] Processing CreateTablet for tablet 4637b961328f435888f41f987fc9aefb (DEFAULT_TABLE table=test-table [id=6c2547635e9943eda418ed85e0c0d7e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:37.841610 30801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4637b961328f435888f41f987fc9aefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:37.841666 30533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4637b961328f435888f41f987fc9aefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:37.841666 30667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4637b961328f435888f41f987fc9aefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:37.847174 30885 tablet_bootstrap.cc:492] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595: Bootstrap starting.
I20260925 02:56:37.848845 30886 tablet_bootstrap.cc:492] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9: Bootstrap starting.
I20260925 02:56:37.849556 30885 tablet_bootstrap.cc:654] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:37.850422 30886 tablet_bootstrap.cc:654] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:37.850556 30885 log.cc:824] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:37.851095 30887 tablet_bootstrap.cc:492] T 4637b961328f435888f41f987fc9aefb P 5fda004723794b4ea19f807f56c8ba61: Bootstrap starting.
I20260925 02:56:37.851533 30886 log.cc:824] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:37.852381 30885 tablet_bootstrap.cc:492] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595: No bootstrap required, opened a new log
I20260925 02:56:37.852568 30885 ts_tablet_manager.cc:1397] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 02:56:37.852844 30887 tablet_bootstrap.cc:654] T 4637b961328f435888f41f987fc9aefb P 5fda004723794b4ea19f807f56c8ba61: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:37.853344 30886 tablet_bootstrap.cc:492] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9: No bootstrap required, opened a new log
I20260925 02:56:37.853592 30886 ts_tablet_manager.cc:1397] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:56:37.854111 30887 log.cc:824] T 4637b961328f435888f41f987fc9aefb P 5fda004723794b4ea19f807f56c8ba61: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:37.855683 30887 tablet_bootstrap.cc:492] T 4637b961328f435888f41f987fc9aefb P 5fda004723794b4ea19f807f56c8ba61: No bootstrap required, opened a new log
I20260925 02:56:37.855851 30887 ts_tablet_manager.cc:1397] T 4637b961328f435888f41f987fc9aefb P 5fda004723794b4ea19f807f56c8ba61: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:56:37.856205 30885 raft_consensus.cc:348] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f1703bc1848fba1ee7274f74ffba9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34699 } } peers { permanent_uuid: "5fda004723794b4ea19f807f56c8ba61" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40009 } } peers { permanent_uuid: "c51fb9333e714a44967490d4bbb96595" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46511 } }
I20260925 02:56:37.856526 30885 raft_consensus.cc:374] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:37.856594 30885 raft_consensus.cc:729] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c51fb9333e714a44967490d4bbb96595, State: Initialized, Role: FOLLOWER
I20260925 02:56:37.857019 30885 consensus_queue.cc:260] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f1703bc1848fba1ee7274f74ffba9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34699 } } peers { permanent_uuid: "5fda004723794b4ea19f807f56c8ba61" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40009 } } peers { permanent_uuid: "c51fb9333e714a44967490d4bbb96595" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46511 } }
I20260925 02:56:37.857154 30886 raft_consensus.cc:348] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f1703bc1848fba1ee7274f74ffba9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34699 } } peers { permanent_uuid: "5fda004723794b4ea19f807f56c8ba61" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40009 } } peers { permanent_uuid: "c51fb9333e714a44967490d4bbb96595" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46511 } }
I20260925 02:56:37.857411 30886 raft_consensus.cc:374] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:37.857468 30886 raft_consensus.cc:729] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95f1703bc1848fba1ee7274f74ffba9, State: Initialized, Role: FOLLOWER
I20260925 02:56:37.857827 30886 consensus_queue.cc:260] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f1703bc1848fba1ee7274f74ffba9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34699 } } peers { permanent_uuid: "5fda004723794b4ea19f807f56c8ba61" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40009 } } peers { permanent_uuid: "c51fb9333e714a44967490d4bbb96595" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46511 } }
I20260925 02:56:37.857928 30885 ts_tablet_manager.cc:1428] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260925 02:56:37.858214 30867 heartbeater.cc:503] Master 127.12.222.254:38031 was elected leader, sending a full tablet report...
I20260925 02:56:37.858423 30733 heartbeater.cc:503] Master 127.12.222.254:38031 was elected leader, sending a full tablet report...
I20260925 02:56:37.858667 30886 ts_tablet_manager.cc:1428] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:56:37.859082 30887 raft_consensus.cc:348] T 4637b961328f435888f41f987fc9aefb P 5fda004723794b4ea19f807f56c8ba61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f1703bc1848fba1ee7274f74ffba9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34699 } } peers { permanent_uuid: "5fda004723794b4ea19f807f56c8ba61" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40009 } } peers { permanent_uuid: "c51fb9333e714a44967490d4bbb96595" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46511 } }
I20260925 02:56:37.859320 30887 raft_consensus.cc:374] T 4637b961328f435888f41f987fc9aefb P 5fda004723794b4ea19f807f56c8ba61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:37.859386 30887 raft_consensus.cc:729] T 4637b961328f435888f41f987fc9aefb P 5fda004723794b4ea19f807f56c8ba61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fda004723794b4ea19f807f56c8ba61, State: Initialized, Role: FOLLOWER
I20260925 02:56:37.859795 30887 consensus_queue.cc:260] T 4637b961328f435888f41f987fc9aefb P 5fda004723794b4ea19f807f56c8ba61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f1703bc1848fba1ee7274f74ffba9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34699 } } peers { permanent_uuid: "5fda004723794b4ea19f807f56c8ba61" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40009 } } peers { permanent_uuid: "c51fb9333e714a44967490d4bbb96595" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46511 } }
I20260925 02:56:37.860391 30599 heartbeater.cc:503] Master 127.12.222.254:38031 was elected leader, sending a full tablet report...
I20260925 02:56:37.861018 30887 ts_tablet_manager.cc:1428] T 4637b961328f435888f41f987fc9aefb P 5fda004723794b4ea19f807f56c8ba61: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260925 02:56:37.889717 30600 tablet.cc:2369] T 4637b961328f435888f41f987fc9aefb P 5fda004723794b4ea19f807f56c8ba61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:38.030434 30892 raft_consensus.cc:482] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:56:38.030673 30892 raft_consensus.cc:504] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f1703bc1848fba1ee7274f74ffba9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34699 } } peers { permanent_uuid: "5fda004723794b4ea19f807f56c8ba61" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40009 } } peers { permanent_uuid: "c51fb9333e714a44967490d4bbb96595" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46511 } }
I20260925 02:56:38.031831 30892 leader_election.cc:288] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fda004723794b4ea19f807f56c8ba61 (127.12.222.193:40009), c51fb9333e714a44967490d4bbb96595 (127.12.222.195:46511)
I20260925 02:56:38.036394 30884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.032150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:46323 (local address 127.12.222.195:46511)
0925 02:56:38.032302 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:38.032306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:38.032663 (+ 357us) server_negotiation.cc:407] Connection header received
0925 02:56:38.032769 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:38.032771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:38.032813 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:38.032906 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:38.033151 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.033856 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.035214 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.035761 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.035874 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.035926 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:38.036205 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 02:56:38.036282 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:56:38.036402 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.032128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:44903 (local address 127.12.222.193:40009)
0925 02:56:38.032294 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:38.032301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:38.032319 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:56:38.032471 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:38.032474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:38.032517 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:38.032584 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:38.032916 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.033745 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.035214 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.035761 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.035874 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.035921 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:38.036205 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 02:56:38.036283 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:56:38.036458 30735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.032025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:40009 (local address 127.12.222.194:44903)
0925 02:56:38.032200 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:38.032214 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:38.032335 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:38.032631 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:38.032635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:38.032673 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:38.032805 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.032810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.033850 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.033852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.035079 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.035089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.035883 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.035885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.036056 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.036101 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 02:56:38.036170 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 02:56:38.036290 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":78,"spinlock_wait_cycles":6016}
I20260925 02:56:38.036466 30894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.032048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:46511 (local address 127.12.222.194:46323)
0925 02:56:38.032534 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:38.032544 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:38.032675 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:38.032907 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:38.032913 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:38.032932 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:38.033021 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.033026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.033961 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.033964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.035084 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.035089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.035880 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.035883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.036036 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.036106 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 02:56:38.036170 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 02:56:38.036291 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":128,"spinlock_wait_cycles":24704,"thread_start_us":100,"threads_started":1}
I20260925 02:56:38.037168 30553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4637b961328f435888f41f987fc9aefb" candidate_uuid: "a95f1703bc1848fba1ee7274f74ffba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fda004723794b4ea19f807f56c8ba61" is_pre_election: true
I20260925 02:56:38.037357 30821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4637b961328f435888f41f987fc9aefb" candidate_uuid: "a95f1703bc1848fba1ee7274f74ffba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c51fb9333e714a44967490d4bbb96595" is_pre_election: true
I20260925 02:56:38.037441 30553 raft_consensus.cc:2454] T 4637b961328f435888f41f987fc9aefb P 5fda004723794b4ea19f807f56c8ba61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a95f1703bc1848fba1ee7274f74ffba9 in term 0.
I20260925 02:56:38.037635 30821 raft_consensus.cc:2454] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a95f1703bc1848fba1ee7274f74ffba9 in term 0.
I20260925 02:56:38.037957 30621 leader_election.cc:302] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fda004723794b4ea19f807f56c8ba61, a95f1703bc1848fba1ee7274f74ffba9; no voters:
I20260925 02:56:38.038245 30892 raft_consensus.cc:2781] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:56:38.038322 30892 raft_consensus.cc:482] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:56:38.038350 30892 raft_consensus.cc:3037] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:38.039274 30892 raft_consensus.cc:504] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f1703bc1848fba1ee7274f74ffba9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34699 } } peers { permanent_uuid: "5fda004723794b4ea19f807f56c8ba61" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40009 } } peers { permanent_uuid: "c51fb9333e714a44967490d4bbb96595" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46511 } }
I20260925 02:56:38.039551 30892 leader_election.cc:288] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [CANDIDATE]: Term 1 election: Requested vote from peers 5fda004723794b4ea19f807f56c8ba61 (127.12.222.193:40009), c51fb9333e714a44967490d4bbb96595 (127.12.222.195:46511)
I20260925 02:56:38.039837 30553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4637b961328f435888f41f987fc9aefb" candidate_uuid: "a95f1703bc1848fba1ee7274f74ffba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fda004723794b4ea19f807f56c8ba61"
I20260925 02:56:38.039853 30821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4637b961328f435888f41f987fc9aefb" candidate_uuid: "a95f1703bc1848fba1ee7274f74ffba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c51fb9333e714a44967490d4bbb96595"
I20260925 02:56:38.039956 30553 raft_consensus.cc:3037] T 4637b961328f435888f41f987fc9aefb P 5fda004723794b4ea19f807f56c8ba61 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:38.039969 30821 raft_consensus.cc:3037] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:38.041021 30821 raft_consensus.cc:2454] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a95f1703bc1848fba1ee7274f74ffba9 in term 1.
I20260925 02:56:38.041029 30553 raft_consensus.cc:2454] T 4637b961328f435888f41f987fc9aefb P 5fda004723794b4ea19f807f56c8ba61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a95f1703bc1848fba1ee7274f74ffba9 in term 1.
I20260925 02:56:38.041296 30620 leader_election.cc:302] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a95f1703bc1848fba1ee7274f74ffba9, c51fb9333e714a44967490d4bbb96595; no voters:
I20260925 02:56:38.041457 30892 raft_consensus.cc:2781] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:38.041662 30892 raft_consensus.cc:686] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [term 1 LEADER]: Becoming Leader. State: Replica: a95f1703bc1848fba1ee7274f74ffba9, State: Running, Role: LEADER
I20260925 02:56:38.041960 30892 consensus_queue.cc:237] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f1703bc1848fba1ee7274f74ffba9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34699 } } peers { permanent_uuid: "5fda004723794b4ea19f807f56c8ba61" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40009 } } peers { permanent_uuid: "c51fb9333e714a44967490d4bbb96595" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46511 } }
I20260925 02:56:38.044930 30407 catalog_manager.cc:5495] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a95f1703bc1848fba1ee7274f74ffba9 (127.12.222.194). New cstate: current_term: 1 leader_uuid: "a95f1703bc1848fba1ee7274f74ffba9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f1703bc1848fba1ee7274f74ffba9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fda004723794b4ea19f807f56c8ba61" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c51fb9333e714a44967490d4bbb96595" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46511 } health_report { overall_health: UNKNOWN } } }
W20260925 02:56:38.045379 30868 tablet.cc:2369] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:38.067693 30881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.064468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.12.222.194:34699)
0925 02:56:38.064577 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:38.064583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:38.064654 (+ 71us) server_negotiation.cc:407] Connection header received
0925 02:56:38.064816 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:38.064823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:38.064869 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:38.064969 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:38.065200 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.065638 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.066413 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.066607 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.066705 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.066997 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:38.067331 (+ 334us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:38.067461 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:38.067524 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 02:56:38.067572 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:56:38.072651 30553 raft_consensus.cc:1261] T 4637b961328f435888f41f987fc9aefb P 5fda004723794b4ea19f807f56c8ba61 [term 1 FOLLOWER]: Refusing update from remote peer a95f1703bc1848fba1ee7274f74ffba9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:38.072651 30821 raft_consensus.cc:1261] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595 [term 1 FOLLOWER]: Refusing update from remote peer a95f1703bc1848fba1ee7274f74ffba9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:38.073330 30895 consensus_queue.cc:1035] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fda004723794b4ea19f807f56c8ba61" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 40009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:38.073488 30892 consensus_queue.cc:1035] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c51fb9333e714a44967490d4bbb96595" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:38.081175 30900 mvcc.cc:205] Tried to move back new op lower bound from 7333089272099360768 to 7333089271982133248. Current Snapshot: MvccSnapshot[applied={T|T < 7333089272099360768}]
I20260925 02:56:38.081967 30899 mvcc.cc:205] Tried to move back new op lower bound from 7333089272099360768 to 7333089271982133248. Current Snapshot: MvccSnapshot[applied={T|T < 7333089272099360768}]
I20260925 02:56:38.086900 30407 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:38.087013 30407 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:38.089005 30407 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f330fa3ca6084e9991de752dfb314ded: Sending DeleteTablet for 3 replicas of tablet 4637b961328f435888f41f987fc9aefb
I20260925 02:56:38.089691 30667 tablet_service.cc:1514] Processing DeleteTablet for tablet 4637b961328f435888f41f987fc9aefb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:38 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:54532
I20260925 02:56:38.089807 30533 tablet_service.cc:1514] Processing DeleteTablet for tablet 4637b961328f435888f41f987fc9aefb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:38 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:51432
I20260925 02:56:38.089830 30801 tablet_service.cc:1514] Processing DeleteTablet for tablet 4637b961328f435888f41f987fc9aefb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:38 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:53936
I20260925 02:56:38.090198 30909 tablet_replica.cc:330] T 4637b961328f435888f41f987fc9aefb P 5fda004723794b4ea19f807f56c8ba61: stopping tablet replica
I20260925 02:56:38.090260 30908 tablet_replica.cc:330] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9: stopping tablet replica
I20260925 02:56:38.090620 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 30470
I20260925 02:56:38.090648 30909 raft_consensus.cc:2229] T 4637b961328f435888f41f987fc9aefb P 5fda004723794b4ea19f807f56c8ba61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:38.091424 30908 raft_consensus.cc:2229] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:56:38.091518 30910 tablet_replica.cc:330] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595: stopping tablet replica
I20260925 02:56:38.091878 30910 raft_consensus.cc:2229] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:38.091877 30908 raft_consensus.cc:2258] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:38.092195 30910 raft_consensus.cc:2258] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:38.093950 30910 ts_tablet_manager.cc:1905] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:38.094195 30908 ts_tablet_manager.cc:1905] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:38.096741 30908 ts_tablet_manager.cc:1918] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:56:38.096864 30908 log.cc:1196] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/4637b961328f435888f41f987fc9aefb
I20260925 02:56:38.097199 30908 ts_tablet_manager.cc:1939] T 4637b961328f435888f41f987fc9aefb P a95f1703bc1848fba1ee7274f74ffba9: Deleting consensus metadata
I20260925 02:56:38.097203 30910 ts_tablet_manager.cc:1918] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:56:38.097328 30910 log.cc:1196] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/4637b961328f435888f41f987fc9aefb
I20260925 02:56:38.097699 30910 ts_tablet_manager.cc:1939] T 4637b961328f435888f41f987fc9aefb P c51fb9333e714a44967490d4bbb96595: Deleting consensus metadata
I20260925 02:56:38.098145 30394 catalog_manager.cc:4841] TS a95f1703bc1848fba1ee7274f74ffba9 (127.12.222.194:34699): tablet 4637b961328f435888f41f987fc9aefb (table test-table [id=6c2547635e9943eda418ed85e0c0d7e2]) successfully deleted
W20260925 02:56:38.098635 30393 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.222.193:40009 (error 108)
I20260925 02:56:38.098650 30392 catalog_manager.cc:4841] TS c51fb9333e714a44967490d4bbb96595 (127.12.222.195:46511): tablet 4637b961328f435888f41f987fc9aefb (table test-table [id=6c2547635e9943eda418ed85e0c0d7e2]) successfully deleted
I20260925 02:56:38.099568 30879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.099274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:40009 (local address 127.0.0.1:51438)
0925 02:56:38.099413 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:38.099481 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.193:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 02:56:38.099663 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 30604
W20260925 02:56:38.099737 30393 catalog_manager.cc:4584] TS 5fda004723794b4ea19f807f56c8ba61 (127.12.222.193:40009): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4637b961328f435888f41f987fc9aefb: Network error: Client connection negotiation failed: client connection to 127.12.222.193:40009: connect: Connection refused (error 111)
I20260925 02:56:38.105278 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 30738
I20260925 02:56:38.110582 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 30376
2026-09-25T02:56:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2238 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T02:56:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:38Z Disabled control of system clock
I20260925 02:56:38.138509 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:43159
--webserver_interface=127.12.222.254
--webserver_port=0
--builtin_ntp_servers=127.12.222.212:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:43159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:56:38.236373 30915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:38.236559 30915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:38.236589 30915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:38.240340 30915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:38.240381 30915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:38.240398 30915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:38.240415 30915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:38.240429 30915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:38.244922 30915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:43159
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:43159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:38.245664 30915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:38.246778 30915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:38.254274 30921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:38.254300 30920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:38.254300 30923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:38.254386 30915 server_base.cc:1034] running on GCE node
I20260925 02:56:38.255039 30915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:38.256017 30915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:38.257191 30915 hybrid_clock.cc:648] HybridClock initialized: now 1790304998257166 us; error 40 us; skew 500 ppm
I20260925 02:56:38.258688 30915 webserver.cc:460] Webserver started at http://127.12.222.254:37347/ using document root <none> and password file <none>
I20260925 02:56:38.259166 30915 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:38.259248 30915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:38.259449 30915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:38.261220 30915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "5d600eaa170f4ac79c2dfe08c2ccef83"
format_stamp: "Formatted at 2026-09-25 02:56:38 on dist-test-slave-252r"
server_key: "f6e7b04b798007293e806875d3aef7f8"
server_key_iv: "c274f5517d42e569d41fd73127d8f80d"
server_key_version: "encryptionkey@0"
I20260925 02:56:38.261729 30915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "5d600eaa170f4ac79c2dfe08c2ccef83"
format_stamp: "Formatted at 2026-09-25 02:56:38 on dist-test-slave-252r"
server_key: "f6e7b04b798007293e806875d3aef7f8"
server_key_iv: "c274f5517d42e569d41fd73127d8f80d"
server_key_version: "encryptionkey@0"
I20260925 02:56:38.265050 30915 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:56:38.267129 30929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:38.268152 30915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:38.268298 30915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "5d600eaa170f4ac79c2dfe08c2ccef83"
format_stamp: "Formatted at 2026-09-25 02:56:38 on dist-test-slave-252r"
server_key: "f6e7b04b798007293e806875d3aef7f8"
server_key_iv: "c274f5517d42e569d41fd73127d8f80d"
server_key_version: "encryptionkey@0"
I20260925 02:56:38.268409 30915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:38.289973 30915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:38.290504 30915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:38.290699 30915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:38.297189 30915 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:43159
I20260925 02:56:38.297216 30982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:43159 every 8 connection(s)
I20260925 02:56:38.297986 30915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:38.300778 30983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:38.303248 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 30915
I20260925 02:56:38.303399 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:56:38.303642 13179 external_mini_cluster.cc:1428] Setting key dccd9a6153aa2d0314aa425ff984ddd2
I20260925 02:56:38.306149 30983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83: Bootstrap starting.
I20260925 02:56:38.308630 30983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:38.309387 30983 log.cc:824] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:38.311470 30983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83: No bootstrap required, opened a new log
I20260925 02:56:38.313258 30986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.304943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.12.222.254:43159)
0925 02:56:38.305557 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:38.305576 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:38.305615 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:56:38.306400 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:38.306422 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:38.306859 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:38.307212 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:38.307626 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.309017 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.309951 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.310536 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.310687 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.311432 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:38.311437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:38.311451 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:38.311897 (+ 446us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:38.311959 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:38.311984 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:38.312071 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:38.312189 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 02:56:38.312434 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":95,"threads_started":1}
I20260925 02:56:38.314877 30983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d600eaa170f4ac79c2dfe08c2ccef83" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43159 } }
I20260925 02:56:38.315061 30983 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:38.315124 30983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d600eaa170f4ac79c2dfe08c2ccef83, State: Initialized, Role: FOLLOWER
I20260925 02:56:38.315553 30983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d600eaa170f4ac79c2dfe08c2ccef83" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43159 } }
I20260925 02:56:38.315667 30983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:38.315747 30983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:38.315860 30983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:38.316707 30983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d600eaa170f4ac79c2dfe08c2ccef83" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43159 } }
I20260925 02:56:38.317018 30983 leader_election.cc:302] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d600eaa170f4ac79c2dfe08c2ccef83; no voters:
I20260925 02:56:38.317266 30983 leader_election.cc:288] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:38.317406 30988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:38.317616 30988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83 [term 1 LEADER]: Becoming Leader. State: Replica: 5d600eaa170f4ac79c2dfe08c2ccef83, State: Running, Role: LEADER
I20260925 02:56:38.317925 30988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d600eaa170f4ac79c2dfe08c2ccef83" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43159 } }
I20260925 02:56:38.318544 30983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:38.319048 30990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d600eaa170f4ac79c2dfe08c2ccef83. Latest consensus state: current_term: 1 leader_uuid: "5d600eaa170f4ac79c2dfe08c2ccef83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d600eaa170f4ac79c2dfe08c2ccef83" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43159 } } }
I20260925 02:56:38.319154 30990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:38.319499 30989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d600eaa170f4ac79c2dfe08c2ccef83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d600eaa170f4ac79c2dfe08c2ccef83" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43159 } } }
I20260925 02:56:38.319597 30989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:38.320742 30993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:38.323230 30993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:38.328341 30993 catalog_manager.cc:1347] Generated new cluster ID: 255d444d6a4443b6a5386b74bbe3fc50
I20260925 02:56:38.328402 30993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:38.335860 30993 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:38.336946 30993 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:38.345198 30993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83: Generated new TSK 0
I20260925 02:56:38.345729 30993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:38.352562 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:43159
--builtin_ntp_servers=127.12.222.212:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:56:38.452872 31007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:38.453095 31007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:38.453171 31007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:38.456808 31007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:38.456848 31007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:38.456931 31007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:56:38.461313 31007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:43159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:38.462183 31007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:38.463366 31007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:38.471788 31012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:38.471758 31015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:38.471837 31007 server_base.cc:1034] running on GCE node
W20260925 02:56:38.471758 31013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:38.472347 31007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:38.472916 31007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:38.474068 31007 hybrid_clock.cc:648] HybridClock initialized: now 1790304998474049 us; error 27 us; skew 500 ppm
I20260925 02:56:38.475677 31007 webserver.cc:460] Webserver started at http://127.12.222.193:44603/ using document root <none> and password file <none>
I20260925 02:56:38.476188 31007 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:38.476339 31007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:38.476552 31007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:38.478444 31007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "e84ee2520d0d4440af8a4883122de777"
format_stamp: "Formatted at 2026-09-25 02:56:38 on dist-test-slave-252r"
server_key: "90e809aefdc68b1a508b54219e381952"
server_key_iv: "a79714e4420d2ac5bd9d5ef36f5429c1"
server_key_version: "encryptionkey@0"
I20260925 02:56:38.478976 31007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "e84ee2520d0d4440af8a4883122de777"
format_stamp: "Formatted at 2026-09-25 02:56:38 on dist-test-slave-252r"
server_key: "90e809aefdc68b1a508b54219e381952"
server_key_iv: "a79714e4420d2ac5bd9d5ef36f5429c1"
server_key_version: "encryptionkey@0"
I20260925 02:56:38.482381 31007 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 02:56:38.484473 31021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:38.485535 31007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:56:38.485622 31007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "e84ee2520d0d4440af8a4883122de777"
format_stamp: "Formatted at 2026-09-25 02:56:38 on dist-test-slave-252r"
server_key: "90e809aefdc68b1a508b54219e381952"
server_key_iv: "a79714e4420d2ac5bd9d5ef36f5429c1"
server_key_version: "encryptionkey@0"
I20260925 02:56:38.485707 31007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:38.506313 31007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:38.506825 31007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:38.506945 31007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:38.507359 31007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:38.508158 31007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:38.508200 31007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:38.508277 31007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:38.508311 31007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:38.515877 31007 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:42079
I20260925 02:56:38.515903 31135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:42079 every 8 connection(s)
I20260925 02:56:38.516654 31007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:56:38.518356 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 31007
I20260925 02:56:38.518479 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:56:38.518774 13179 external_mini_cluster.cc:1428] Setting key bac22384d7eca1307aa17e0bb4123378
I20260925 02:56:38.520903 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:43159
--builtin_ntp_servers=127.12.222.212:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:56:38.531270 30986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.518236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:44973 (local address 127.12.222.254:43159)
0925 02:56:38.518384 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:38.518389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:38.519527 (+ 1138us) server_negotiation.cc:407] Connection header received
0925 02:56:38.521269 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:38.521272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:38.521315 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:38.521374 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:38.522425 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.523217 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.524323 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.524564 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.524638 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.530556 (+ 5918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:38.530559 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:38.530561 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:38.530572 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:38.530589 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:38.530596 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:38.530650 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:38.531040 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 02:56:38.531093 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:56:38.532215 31138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.518651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43159 (local address 127.12.222.193:44973)
0925 02:56:38.519352 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:38.519401 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:38.521052 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:38.521536 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:38.521547 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:38.521646 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:38.522269 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.522290 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.523345 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.523348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.524151 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.524159 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.524708 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.524715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.524934 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.530242 (+ 5308us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:56:38.530246 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:38.530301 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:56:38.530386 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:38.530682 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:38.530694 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:38.530703 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 02:56:38.530845 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 02:56:38.531175 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":139,"threads_started":1}
I20260925 02:56:38.533481 31136 heartbeater.cc:347] Connected to a master server at 127.12.222.254:43159
I20260925 02:56:38.533703 31136 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:38.534219 31136 heartbeater.cc:511] Master 127.12.222.254:43159 requested a full tablet report, sending...
I20260925 02:56:38.535580 30946 ts_manager.cc:194] Registered new tserver with Master: e84ee2520d0d4440af8a4883122de777 (127.12.222.193:42079)
I20260925 02:56:38.536761 30946 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.222.193:44973
W20260925 02:56:38.625699 31139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:38.625890 31139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:38.625922 31139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:38.629689 31139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:38.629729 31139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:38.629776 31139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:56:38.634104 31139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:43159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:38.634908 31139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:38.636005 31139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:38.644534 31147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:38.644709 31139 server_base.cc:1034] running on GCE node
W20260925 02:56:38.644543 31144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:38.644834 31145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:38.645164 31139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:38.645722 31139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:38.646885 31139 hybrid_clock.cc:648] HybridClock initialized: now 1790304998646868 us; error 27 us; skew 500 ppm
I20260925 02:56:38.648583 31139 webserver.cc:460] Webserver started at http://127.12.222.194:33787/ using document root <none> and password file <none>
I20260925 02:56:38.649070 31139 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:38.649127 31139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:38.649348 31139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:38.651058 31139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "3cc4fc64aaef474a96f668fd54106df9"
format_stamp: "Formatted at 2026-09-25 02:56:38 on dist-test-slave-252r"
server_key: "5f4bc4bdbb76e17f8209da695717a0bd"
server_key_iv: "6c0f7e27b2f593b3c4ddd23956ae2af4"
server_key_version: "encryptionkey@0"
I20260925 02:56:38.651592 31139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "3cc4fc64aaef474a96f668fd54106df9"
format_stamp: "Formatted at 2026-09-25 02:56:38 on dist-test-slave-252r"
server_key: "5f4bc4bdbb76e17f8209da695717a0bd"
server_key_iv: "6c0f7e27b2f593b3c4ddd23956ae2af4"
server_key_version: "encryptionkey@0"
I20260925 02:56:38.655146 31139 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:56:38.657315 31153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:38.658371 31139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:56:38.658506 31139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "3cc4fc64aaef474a96f668fd54106df9"
format_stamp: "Formatted at 2026-09-25 02:56:38 on dist-test-slave-252r"
server_key: "5f4bc4bdbb76e17f8209da695717a0bd"
server_key_iv: "6c0f7e27b2f593b3c4ddd23956ae2af4"
server_key_version: "encryptionkey@0"
I20260925 02:56:38.658607 31139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:38.699370 31139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:38.700202 31139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:38.700457 31139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:38.701246 31139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:38.702558 31139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:38.702625 31139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:38.702688 31139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:38.702723 31139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:38.711750 31139 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:39925
I20260925 02:56:38.711810 31267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:39925 every 8 connection(s)
I20260925 02:56:38.712673 31139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:56:38.718355 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 31139
I20260925 02:56:38.718502 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:56:38.718768 13179 external_mini_cluster.cc:1428] Setting key 7561ee97915ccb55a823f0437d3d8a97
I20260925 02:56:38.720858 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:43159
--builtin_ntp_servers=127.12.222.212:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:56:38.721771 30986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.714323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:33509 (local address 127.12.222.254:43159)
0925 02:56:38.714441 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:38.714445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:38.715340 (+ 895us) server_negotiation.cc:407] Connection header received
0925 02:56:38.716756 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:38.716759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:38.716830 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:38.716913 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:38.717694 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.718293 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.719764 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.720071 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.720184 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.721297 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:38.721300 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:38.721303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:38.721314 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:38.721335 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:38.721350 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:38.721423 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:38.721580 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 02:56:38.721637 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:56:38.722802 31270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.714636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43159 (local address 127.12.222.194:33509)
0925 02:56:38.715204 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:38.715240 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:38.716543 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:38.717055 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:38.717061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:38.717134 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:38.717538 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.717550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.718434 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.718437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.719643 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.719654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.720220 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.720224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.720413 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.721047 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:56:38.721051 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:38.721098 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:56:38.721154 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:38.721441 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:38.721449 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:38.721455 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:56:38.721579 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 02:56:38.721830 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":92,"threads_started":1}
I20260925 02:56:38.723906 31268 heartbeater.cc:347] Connected to a master server at 127.12.222.254:43159
I20260925 02:56:38.724126 31268 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:38.724828 31268 heartbeater.cc:511] Master 127.12.222.254:43159 requested a full tablet report, sending...
I20260925 02:56:38.725915 30946 ts_manager.cc:194] Registered new tserver with Master: 3cc4fc64aaef474a96f668fd54106df9 (127.12.222.194:39925)
I20260925 02:56:38.726461 30946 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.222.194:33509
W20260925 02:56:38.824713 31271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:38.824939 31271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:38.824971 31271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:38.828497 31271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:38.828536 31271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:38.828611 31271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:56:38.833455 31271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:43159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:38.834348 31271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:38.835507 31271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:38.844525 31279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:38.844585 31276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:38.844554 31277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:38.844956 31271 server_base.cc:1034] running on GCE node
I20260925 02:56:38.845331 31271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:38.845924 31271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:38.847100 31271 hybrid_clock.cc:648] HybridClock initialized: now 1790304998847066 us; error 52 us; skew 500 ppm
I20260925 02:56:38.848732 31271 webserver.cc:460] Webserver started at http://127.12.222.195:35427/ using document root <none> and password file <none>
I20260925 02:56:38.849270 31271 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:38.849352 31271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:38.849565 31271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:38.851470 31271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "b85d552dc05246b69e5d80ab3ecaeaed"
format_stamp: "Formatted at 2026-09-25 02:56:38 on dist-test-slave-252r"
server_key: "6a94df5a0f4f319e5a9625085a0b3f07"
server_key_iv: "1ab9fccfa48afbbcb5dff34d339941d0"
server_key_version: "encryptionkey@0"
I20260925 02:56:38.852000 31271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "b85d552dc05246b69e5d80ab3ecaeaed"
format_stamp: "Formatted at 2026-09-25 02:56:38 on dist-test-slave-252r"
server_key: "6a94df5a0f4f319e5a9625085a0b3f07"
server_key_iv: "1ab9fccfa48afbbcb5dff34d339941d0"
server_key_version: "encryptionkey@0"
I20260925 02:56:38.855382 31271 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:56:38.857520 31285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:38.858534 31271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:56:38.858690 31271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "b85d552dc05246b69e5d80ab3ecaeaed"
format_stamp: "Formatted at 2026-09-25 02:56:38 on dist-test-slave-252r"
server_key: "6a94df5a0f4f319e5a9625085a0b3f07"
server_key_iv: "1ab9fccfa48afbbcb5dff34d339941d0"
server_key_version: "encryptionkey@0"
I20260925 02:56:38.858805 31271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:38.874107 31271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:38.874676 31271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:38.874863 31271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:38.875372 31271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:38.876191 31271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:38.876305 31271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:38.876364 31271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:38.876437 31271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:38.884545 31271 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:33233
I20260925 02:56:38.884574 31399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:33233 every 8 connection(s)
I20260925 02:56:38.885313 31271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:56:38.887471 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 31271
I20260925 02:56:38.887570 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:56:38.887827 13179 external_mini_cluster.cc:1428] Setting key 40bef57025651bb470bc0f227021152d
I20260925 02:56:38.893905 30986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.886806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:43809 (local address 127.12.222.254:43159)
0925 02:56:38.886977 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:38.886982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:38.888017 (+ 1035us) server_negotiation.cc:407] Connection header received
0925 02:56:38.889616 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:38.889619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:38.889686 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:38.889752 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:38.890604 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.891083 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.892103 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.892363 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.892465 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.893412 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:38.893415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:38.893418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:38.893430 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:38.893453 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:38.893467 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:38.893541 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:38.893705 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 02:56:38.893763 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 02:56:38.894631 31402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.887187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:43159 (local address 127.12.222.195:43809)
0925 02:56:38.887848 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:38.887913 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:38.889434 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:38.889952 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:38.889960 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:38.890066 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:38.890471 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.890482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.891203 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.891206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.891983 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.891992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.892500 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.892504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.892666 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.893171 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:56:38.893172 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:38.893208 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:56:38.893263 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:38.893555 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:38.893566 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:38.893579 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 02:56:38.893700 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 02:56:38.893947 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":133,"threads_started":1}
I20260925 02:56:38.895593 31400 heartbeater.cc:347] Connected to a master server at 127.12.222.254:43159
I20260925 02:56:38.895783 31400 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:38.896198 31400 heartbeater.cc:511] Master 127.12.222.254:43159 requested a full tablet report, sending...
I20260925 02:56:38.897116 30946 ts_manager.cc:194] Registered new tserver with Master: b85d552dc05246b69e5d80ab3ecaeaed (127.12.222.195:33233)
I20260925 02:56:38.897626 30946 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.222.195:43809
I20260925 02:56:38.901484 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:56:38.906476 30986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.903261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59866 (local address 127.12.222.254:43159)
0925 02:56:38.903386 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:38.903391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:38.903541 (+ 150us) server_negotiation.cc:407] Connection header received
0925 02:56:38.903627 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:38.903629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:38.903662 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:38.903722 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:38.903951 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.904463 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.905191 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.905391 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.905457 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.906128 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:38.906131 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:38.906133 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:38.906143 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:38.906158 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:38.906165 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:38.906220 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:38.906290 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 02:56:38.906327 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:56:38.908769 30946 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:56:38.911430 30946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:56:38.924168 31411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.918531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:42079 (local address 127.0.0.1:40970)
0925 02:56:38.918817 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:38.918860 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:38.918979 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:38.919626 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:38.919631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:38.919653 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:38.919787 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.919795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.921368 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.921371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.922678 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.922688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.923668 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.923674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.923789 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.923827 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:56:38.923930 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 02:56:38.924044 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":53,"thread_start_us":68,"threads_started":1}
I20260925 02:56:38.924263 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.918646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40970 (local address 127.12.222.193:42079)
0925 02:56:38.919063 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:38.919077 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:38.919113 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:56:38.919185 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:38.919193 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:38.919444 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:38.919940 (+ 496us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:38.920089 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.921252 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.922780 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.923542 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.923662 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.923806 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:38.923972 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 02:56:38.924079 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":103,"threads_started":1}
I20260925 02:56:38.924479 31415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.919425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:39925 (local address 127.0.0.1:46778)
0925 02:56:38.920302 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:38.920314 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:38.920402 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:38.920857 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:38.920860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:38.920876 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:38.920962 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.920970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.922199 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.922202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.923040 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.923047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.924049 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.924053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.924136 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.924156 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 02:56:38.924247 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 02:56:38.924334 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":74,"threads_started":1}
I20260925 02:56:38.924500 31414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.919261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.12.222.195:33233)
0925 02:56:38.919584 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:38.919594 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:38.919640 (+ 46us) server_negotiation.cc:407] Connection header received
0925 02:56:38.919822 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:38.919826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:38.919997 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:38.920171 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:38.920535 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.921525 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.922816 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.923542 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.923662 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.923806 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:38.924228 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 02:56:38.924350 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":66,"threads_started":1}
I20260925 02:56:38.924731 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.919151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:33233 (local address 127.0.0.1:50860)
0925 02:56:38.919570 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:38.919582 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:38.919681 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:38.920302 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:38.920304 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:38.920319 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:38.920423 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.920448 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.921640 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.921643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.922694 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:38.922704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.923886 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.923890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:38.924000 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.924021 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 02:56:38.924090 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 02:56:38.924158 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":71,"thread_start_us":83,"threads_started":1}
I20260925 02:56:38.924862 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:38.919924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46778 (local address 127.12.222.194:39925)
0925 02:56:38.920261 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:38.920276 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:38.920405 (+ 129us) server_negotiation.cc:407] Connection header received
0925 02:56:38.920549 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:38.920554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:38.920689 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:38.920864 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:38.921051 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.922080 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.923163 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:38.923709 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:38.923818 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:38.923987 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:38.924617 (+ 630us) server_negotiation.cc:299] Negotiation successful
0925 02:56:38.924709 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":121,"threads_started":1}
I20260925 02:56:38.927214 31069 tablet_service.cc:1467] Processing CreateTablet for tablet 1af02a09775a481f9225bf91f7be2333 (DEFAULT_TABLE table=test-table [id=4026c6dbaeec4f78bf4e764903e3d5eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:38.927322 31333 tablet_service.cc:1467] Processing CreateTablet for tablet 1af02a09775a481f9225bf91f7be2333 (DEFAULT_TABLE table=test-table [id=4026c6dbaeec4f78bf4e764903e3d5eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:38.927325 31201 tablet_service.cc:1467] Processing CreateTablet for tablet 1af02a09775a481f9225bf91f7be2333 (DEFAULT_TABLE table=test-table [id=4026c6dbaeec4f78bf4e764903e3d5eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:38.928094 31069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1af02a09775a481f9225bf91f7be2333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:38.928208 31333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1af02a09775a481f9225bf91f7be2333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:38.928208 31201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1af02a09775a481f9225bf91f7be2333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:38.933403 31417 tablet_bootstrap.cc:492] T 1af02a09775a481f9225bf91f7be2333 P e84ee2520d0d4440af8a4883122de777: Bootstrap starting.
I20260925 02:56:38.935052 31418 tablet_bootstrap.cc:492] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9: Bootstrap starting.
I20260925 02:56:38.936143 31417 tablet_bootstrap.cc:654] T 1af02a09775a481f9225bf91f7be2333 P e84ee2520d0d4440af8a4883122de777: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:38.936501 31419 tablet_bootstrap.cc:492] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed: Bootstrap starting.
I20260925 02:56:38.936898 31418 tablet_bootstrap.cc:654] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:38.936939 31417 log.cc:824] T 1af02a09775a481f9225bf91f7be2333 P e84ee2520d0d4440af8a4883122de777: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:38.937578 31418 log.cc:824] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:38.938239 31419 tablet_bootstrap.cc:654] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:38.938907 31417 tablet_bootstrap.cc:492] T 1af02a09775a481f9225bf91f7be2333 P e84ee2520d0d4440af8a4883122de777: No bootstrap required, opened a new log
I20260925 02:56:38.938915 31419 log.cc:824] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:38.939077 31417 ts_tablet_manager.cc:1397] T 1af02a09775a481f9225bf91f7be2333 P e84ee2520d0d4440af8a4883122de777: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 02:56:38.939894 31418 tablet_bootstrap.cc:492] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9: No bootstrap required, opened a new log
I20260925 02:56:38.940061 31418 ts_tablet_manager.cc:1397] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:56:38.941442 31419 tablet_bootstrap.cc:492] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed: No bootstrap required, opened a new log
I20260925 02:56:38.941624 31419 ts_tablet_manager.cc:1397] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:56:38.943207 31417 raft_consensus.cc:348] T 1af02a09775a481f9225bf91f7be2333 P e84ee2520d0d4440af8a4883122de777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84ee2520d0d4440af8a4883122de777" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42079 } } peers { permanent_uuid: "b85d552dc05246b69e5d80ab3ecaeaed" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 33233 } } peers { permanent_uuid: "3cc4fc64aaef474a96f668fd54106df9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 39925 } }
I20260925 02:56:38.943462 31417 raft_consensus.cc:374] T 1af02a09775a481f9225bf91f7be2333 P e84ee2520d0d4440af8a4883122de777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:38.943529 31417 raft_consensus.cc:729] T 1af02a09775a481f9225bf91f7be2333 P e84ee2520d0d4440af8a4883122de777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e84ee2520d0d4440af8a4883122de777, State: Initialized, Role: FOLLOWER
I20260925 02:56:38.943776 31418 raft_consensus.cc:348] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84ee2520d0d4440af8a4883122de777" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42079 } } peers { permanent_uuid: "b85d552dc05246b69e5d80ab3ecaeaed" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 33233 } } peers { permanent_uuid: "3cc4fc64aaef474a96f668fd54106df9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 39925 } }
I20260925 02:56:38.944005 31418 raft_consensus.cc:374] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:38.943965 31417 consensus_queue.cc:260] T 1af02a09775a481f9225bf91f7be2333 P e84ee2520d0d4440af8a4883122de777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84ee2520d0d4440af8a4883122de777" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42079 } } peers { permanent_uuid: "b85d552dc05246b69e5d80ab3ecaeaed" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 33233 } } peers { permanent_uuid: "3cc4fc64aaef474a96f668fd54106df9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 39925 } }
I20260925 02:56:38.944056 31418 raft_consensus.cc:729] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc4fc64aaef474a96f668fd54106df9, State: Initialized, Role: FOLLOWER
I20260925 02:56:38.944497 31418 consensus_queue.cc:260] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84ee2520d0d4440af8a4883122de777" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42079 } } peers { permanent_uuid: "b85d552dc05246b69e5d80ab3ecaeaed" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 33233 } } peers { permanent_uuid: "3cc4fc64aaef474a96f668fd54106df9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 39925 } }
I20260925 02:56:38.944968 31417 ts_tablet_manager.cc:1428] T 1af02a09775a481f9225bf91f7be2333 P e84ee2520d0d4440af8a4883122de777: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260925 02:56:38.945284 31136 heartbeater.cc:503] Master 127.12.222.254:43159 was elected leader, sending a full tablet report...
I20260925 02:56:38.945358 31418 ts_tablet_manager.cc:1428] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9: Time spent starting tablet: real 0.005s user 0.000s sys 0.007s
I20260925 02:56:38.945286 31419 raft_consensus.cc:348] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84ee2520d0d4440af8a4883122de777" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42079 } } peers { permanent_uuid: "b85d552dc05246b69e5d80ab3ecaeaed" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 33233 } } peers { permanent_uuid: "3cc4fc64aaef474a96f668fd54106df9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 39925 } }
I20260925 02:56:38.945534 31419 raft_consensus.cc:374] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:38.945583 31419 raft_consensus.cc:729] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b85d552dc05246b69e5d80ab3ecaeaed, State: Initialized, Role: FOLLOWER
I20260925 02:56:38.945793 31268 heartbeater.cc:503] Master 127.12.222.254:43159 was elected leader, sending a full tablet report...
I20260925 02:56:38.945950 31419 consensus_queue.cc:260] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84ee2520d0d4440af8a4883122de777" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42079 } } peers { permanent_uuid: "b85d552dc05246b69e5d80ab3ecaeaed" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 33233 } } peers { permanent_uuid: "3cc4fc64aaef474a96f668fd54106df9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 39925 } }
I20260925 02:56:38.946494 31400 heartbeater.cc:503] Master 127.12.222.254:43159 was elected leader, sending a full tablet report...
I20260925 02:56:38.946825 31419 ts_tablet_manager.cc:1428] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260925 02:56:38.963470 31269 tablet.cc:2369] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:56:39.017632 31137 tablet.cc:2369] T 1af02a09775a481f9225bf91f7be2333 P e84ee2520d0d4440af8a4883122de777: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:56:39.136245 31401 tablet.cc:2369] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:39.138265 31424 raft_consensus.cc:482] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:56:39.138412 31424 raft_consensus.cc:504] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84ee2520d0d4440af8a4883122de777" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42079 } } peers { permanent_uuid: "b85d552dc05246b69e5d80ab3ecaeaed" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 33233 } } peers { permanent_uuid: "3cc4fc64aaef474a96f668fd54106df9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 39925 } }
I20260925 02:56:39.139431 31424 leader_election.cc:288] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e84ee2520d0d4440af8a4883122de777 (127.12.222.193:42079), b85d552dc05246b69e5d80ab3ecaeaed (127.12.222.195:33233)
I20260925 02:56:39.144109 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.139793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:49713 (local address 127.12.222.193:42079)
0925 02:56:39.139953 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:39.139957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:39.139973 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:56:39.140128 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:39.140133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:39.140190 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:39.140295 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:39.140593 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.141402 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.142879 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.143465 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.143583 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.143628 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:39.143927 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 02:56:39.143990 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:56:39.144107 31414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.140102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:37851 (local address 127.12.222.195:33233)
0925 02:56:39.140263 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:39.140269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:39.140427 (+ 158us) server_negotiation.cc:407] Connection header received
0925 02:56:39.140514 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:39.140516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:39.140576 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:39.140648 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:39.140916 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.141579 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.142882 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.143465 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.143583 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.143628 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:39.143927 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 02:56:39.143990 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:56:39.144115 31270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.139704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:42079 (local address 127.12.222.194:49713)
0925 02:56:39.139858 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:39.139898 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:39.140009 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:39.140316 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:39.140319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:39.140337 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:39.140483 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:39.140493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.141505 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.141507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:39.142755 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:39.142766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.143591 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.143594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:39.143749 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.143815 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 02:56:39.143886 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 02:56:39.143970 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":80}
I20260925 02:56:39.144124 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.139974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:33233 (local address 127.12.222.194:37851)
0925 02:56:39.140334 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:39.140347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:39.140435 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:39.140670 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:39.140672 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:39.140686 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:39.140785 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:39.140790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.141667 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.141670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:39.142767 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:39.142774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.143590 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.143594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:39.143766 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.143809 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 02:56:39.143886 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:56:39.143965 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":102,"thread_start_us":121,"threads_started":1}
I20260925 02:56:39.144759 31090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af02a09775a481f9225bf91f7be2333" candidate_uuid: "3cc4fc64aaef474a96f668fd54106df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e84ee2520d0d4440af8a4883122de777" is_pre_election: true
I20260925 02:56:39.144945 31353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af02a09775a481f9225bf91f7be2333" candidate_uuid: "3cc4fc64aaef474a96f668fd54106df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b85d552dc05246b69e5d80ab3ecaeaed" is_pre_election: true
I20260925 02:56:39.145022 31090 raft_consensus.cc:2454] T 1af02a09775a481f9225bf91f7be2333 P e84ee2520d0d4440af8a4883122de777 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cc4fc64aaef474a96f668fd54106df9 in term 0.
I20260925 02:56:39.145192 31353 raft_consensus.cc:2454] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cc4fc64aaef474a96f668fd54106df9 in term 0.
I20260925 02:56:39.145493 31156 leader_election.cc:302] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cc4fc64aaef474a96f668fd54106df9, e84ee2520d0d4440af8a4883122de777; no voters:
I20260925 02:56:39.145790 31424 raft_consensus.cc:2781] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:56:39.145905 31424 raft_consensus.cc:482] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:56:39.145983 31424 raft_consensus.cc:3037] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:39.146963 31424 raft_consensus.cc:504] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84ee2520d0d4440af8a4883122de777" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42079 } } peers { permanent_uuid: "b85d552dc05246b69e5d80ab3ecaeaed" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 33233 } } peers { permanent_uuid: "3cc4fc64aaef474a96f668fd54106df9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 39925 } }
I20260925 02:56:39.147250 31424 leader_election.cc:288] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [CANDIDATE]: Term 1 election: Requested vote from peers e84ee2520d0d4440af8a4883122de777 (127.12.222.193:42079), b85d552dc05246b69e5d80ab3ecaeaed (127.12.222.195:33233)
I20260925 02:56:39.147522 31090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af02a09775a481f9225bf91f7be2333" candidate_uuid: "3cc4fc64aaef474a96f668fd54106df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e84ee2520d0d4440af8a4883122de777"
I20260925 02:56:39.147624 31353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af02a09775a481f9225bf91f7be2333" candidate_uuid: "3cc4fc64aaef474a96f668fd54106df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b85d552dc05246b69e5d80ab3ecaeaed"
I20260925 02:56:39.147652 31090 raft_consensus.cc:3037] T 1af02a09775a481f9225bf91f7be2333 P e84ee2520d0d4440af8a4883122de777 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:39.147739 31353 raft_consensus.cc:3037] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:39.148900 31353 raft_consensus.cc:2454] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cc4fc64aaef474a96f668fd54106df9 in term 1.
I20260925 02:56:39.148907 31090 raft_consensus.cc:2454] T 1af02a09775a481f9225bf91f7be2333 P e84ee2520d0d4440af8a4883122de777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cc4fc64aaef474a96f668fd54106df9 in term 1.
I20260925 02:56:39.149170 31156 leader_election.cc:302] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cc4fc64aaef474a96f668fd54106df9, b85d552dc05246b69e5d80ab3ecaeaed; no voters:
I20260925 02:56:39.149327 31424 raft_consensus.cc:2781] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:39.149530 31424 raft_consensus.cc:686] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [term 1 LEADER]: Becoming Leader. State: Replica: 3cc4fc64aaef474a96f668fd54106df9, State: Running, Role: LEADER
I20260925 02:56:39.149847 31424 consensus_queue.cc:237] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84ee2520d0d4440af8a4883122de777" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42079 } } peers { permanent_uuid: "b85d552dc05246b69e5d80ab3ecaeaed" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 33233 } } peers { permanent_uuid: "3cc4fc64aaef474a96f668fd54106df9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 39925 } }
I20260925 02:56:39.152863 30945 catalog_manager.cc:5495] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cc4fc64aaef474a96f668fd54106df9 (127.12.222.194). New cstate: current_term: 1 leader_uuid: "3cc4fc64aaef474a96f668fd54106df9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84ee2520d0d4440af8a4883122de777" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b85d552dc05246b69e5d80ab3ecaeaed" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 33233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cc4fc64aaef474a96f668fd54106df9" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 39925 } health_report { overall_health: HEALTHY } } }
I20260925 02:56:39.215950 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.212886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.12.222.194:39925)
0925 02:56:39.213016 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:39.213020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:39.213032 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:56:39.213115 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:39.213117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:39.213171 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:39.213251 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:39.213468 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.213950 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.214790 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:39.214987 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:39.215057 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:39.215292 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:39.215663 (+ 371us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:39.215757 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:39.215808 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 02:56:39.215848 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:56:39.221381 31353 raft_consensus.cc:1261] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed [term 1 FOLLOWER]: Refusing update from remote peer 3cc4fc64aaef474a96f668fd54106df9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:39.221416 31090 raft_consensus.cc:1261] T 1af02a09775a481f9225bf91f7be2333 P e84ee2520d0d4440af8a4883122de777 [term 1 FOLLOWER]: Refusing update from remote peer 3cc4fc64aaef474a96f668fd54106df9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:39.222038 31424 consensus_queue.cc:1035] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85d552dc05246b69e5d80ab3ecaeaed" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 33233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:39.222209 31427 consensus_queue.cc:1035] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e84ee2520d0d4440af8a4883122de777" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:39.229379 31432 mvcc.cc:205] Tried to move back new op lower bound from 7333089276803891200 to 7333089276520169472. Current Snapshot: MvccSnapshot[applied={T|T < 7333089276803891200}]
I20260925 02:56:39.230175 31431 mvcc.cc:205] Tried to move back new op lower bound from 7333089276803891200 to 7333089276520169472. Current Snapshot: MvccSnapshot[applied={T|T < 7333089276803891200}]
I20260925 02:56:39.228842 31430 mvcc.cc:205] Tried to move back new op lower bound from 7333089276803891200 to 7333089276520169472. Current Snapshot: MvccSnapshot[applied={T|T < 7333089276803891200}]
I20260925 02:56:39.237958 30945 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:39.238077 30945 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:39.240159 30945 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d600eaa170f4ac79c2dfe08c2ccef83: Sending DeleteTablet for 3 replicas of tablet 1af02a09775a481f9225bf91f7be2333
I20260925 02:56:39.240855 31069 tablet_service.cc:1514] Processing DeleteTablet for tablet 1af02a09775a481f9225bf91f7be2333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:39 UTC) from {username='slave'} at 127.0.0.1:40970
I20260925 02:56:39.240964 31333 tablet_service.cc:1514] Processing DeleteTablet for tablet 1af02a09775a481f9225bf91f7be2333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:39 UTC) from {username='slave'} at 127.0.0.1:50860
I20260925 02:56:39.241055 31201 tablet_service.cc:1514] Processing DeleteTablet for tablet 1af02a09775a481f9225bf91f7be2333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:39 UTC) from {username='slave'} at 127.0.0.1:46778
I20260925 02:56:39.241343 31440 tablet_replica.cc:330] T 1af02a09775a481f9225bf91f7be2333 P e84ee2520d0d4440af8a4883122de777: stopping tablet replica
I20260925 02:56:39.241523 31441 tablet_replica.cc:330] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed: stopping tablet replica
I20260925 02:56:39.241781 31441 raft_consensus.cc:2229] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:39.241931 31440 raft_consensus.cc:2229] T 1af02a09775a481f9225bf91f7be2333 P e84ee2520d0d4440af8a4883122de777 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:39.242061 31442 tablet_replica.cc:330] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9: stopping tablet replica
I20260925 02:56:39.242156 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 31007
I20260925 02:56:39.242314 31442 raft_consensus.cc:2229] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:56:39.242596 31442 raft_consensus.cc:2258] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:39.244254 31441 raft_consensus.cc:2258] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:39.245111 31442 ts_tablet_manager.cc:1905] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:39.246157 31441 ts_tablet_manager.cc:1905] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:39.247881 31442 ts_tablet_manager.cc:1918] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:56:39.248057 31442 log.cc:1196] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/1af02a09775a481f9225bf91f7be2333
I20260925 02:56:39.248467 31442 ts_tablet_manager.cc:1939] T 1af02a09775a481f9225bf91f7be2333 P 3cc4fc64aaef474a96f668fd54106df9: Deleting consensus metadata
I20260925 02:56:39.248924 31441 ts_tablet_manager.cc:1918] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:56:39.249022 31441 log.cc:1196] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/1af02a09775a481f9225bf91f7be2333
I20260925 02:56:39.249316 31441 ts_tablet_manager.cc:1939] T 1af02a09775a481f9225bf91f7be2333 P b85d552dc05246b69e5d80ab3ecaeaed: Deleting consensus metadata
I20260925 02:56:39.249846 30932 catalog_manager.cc:4841] TS 3cc4fc64aaef474a96f668fd54106df9 (127.12.222.194:39925): tablet 1af02a09775a481f9225bf91f7be2333 (table test-table [id=4026c6dbaeec4f78bf4e764903e3d5eb]) successfully deleted
I20260925 02:56:39.250414 30932 catalog_manager.cc:4841] TS b85d552dc05246b69e5d80ab3ecaeaed (127.12.222.195:33233): tablet 1af02a09775a481f9225bf91f7be2333 (table test-table [id=4026c6dbaeec4f78bf4e764903e3d5eb]) successfully deleted
W20260925 02:56:39.250622 30932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.222.193:42079 (error 108)
I20260925 02:56:39.250866 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 31139
I20260925 02:56:39.252035 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:39.251761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:42079 (local address 127.0.0.1:40980)
0925 02:56:39.251901 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:39.251961 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.193:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260925 02:56:39.252184 30932 catalog_manager.cc:4584] TS e84ee2520d0d4440af8a4883122de777 (127.12.222.193:42079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1af02a09775a481f9225bf91f7be2333: Network error: Client connection negotiation failed: client connection to 127.12.222.193:42079: connect: Connection refused (error 111)
I20260925 02:56:39.256685 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 31271
I20260925 02:56:39.262012 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 30915
2026-09-25T02:56:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1150 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:39 dist-test-slave-252r krb5kdc[31446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:39 dist-test-slave-252r krb5kdc[31446](info): set up 2 sockets
Sep 25 02:56:39 dist-test-slave-252r krb5kdc[31446](info): commencing operation
krb5kdc: starting...
W20260925 02:56:40.244089 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:56:40 dist-test-slave-252r krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:56:40.349277 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:45887
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:45887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:40.448370 31459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:40.448593 31459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:40.448658 31459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:40.452008 31459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:40.452076 31459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:40.452111 31459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:40.452178 31459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:40.452209 31459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:40.456933 31459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:45887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:45887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:40.457690 31459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:40.458474 31459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:40.466073 31467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:40.466041 31464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:40.466230 31459 server_base.cc:1034] running on GCE node
W20260925 02:56:40.466050 31465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:40.466710 31459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:40.467666 31459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:40.468842 31459 hybrid_clock.cc:648] HybridClock initialized: now 1790305000468826 us; error 27 us; skew 500 ppm
Sep 25 02:56:40 dist-test-slave-252r krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305000, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:40.471386 31459 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:56:40.472184 31459 webserver.cc:460] Webserver started at http://127.12.222.254:38393/ using document root <none> and password file <none>
I20260925 02:56:40.472694 31459 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:40.472782 31459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:40.472999 31459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:40.474795 31459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "c0f7eb01f96d4173b58d70b57ba728c5"
format_stamp: "Formatted at 2026-09-25 02:56:40 on dist-test-slave-252r"
server_key: "e3ccc671abca2d7762710ad78009b82a"
server_key_iv: "f80c31ffb45ff304c1628b6700dbf398"
server_key_version: "encryptionkey@0"
I20260925 02:56:40.475314 31459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "c0f7eb01f96d4173b58d70b57ba728c5"
format_stamp: "Formatted at 2026-09-25 02:56:40 on dist-test-slave-252r"
server_key: "e3ccc671abca2d7762710ad78009b82a"
server_key_iv: "f80c31ffb45ff304c1628b6700dbf398"
server_key_version: "encryptionkey@0"
I20260925 02:56:40.478776 31459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 02:56:40.480914 31474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:40.482016 31459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:40.482149 31459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "c0f7eb01f96d4173b58d70b57ba728c5"
format_stamp: "Formatted at 2026-09-25 02:56:40 on dist-test-slave-252r"
server_key: "e3ccc671abca2d7762710ad78009b82a"
server_key_iv: "f80c31ffb45ff304c1628b6700dbf398"
server_key_version: "encryptionkey@0"
I20260925 02:56:40.482246 31459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:40.504208 31459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:40.506682 31459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:40.506855 31459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:40.513355 31459 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:45887
I20260925 02:56:40.513368 31527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:45887 every 8 connection(s)
I20260925 02:56:40.514178 31459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:40.514487 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 31459
I20260925 02:56:40.514627 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:56:40.514902 13179 external_mini_cluster.cc:1428] Setting key c9e6ec5b81e0075d485b20fdaa239200
I20260925 02:56:40.517011 31528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:40.522058 31528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5: Bootstrap starting.
Sep 25 02:56:40 dist-test-slave-252r krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:56:40.524197 31528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:40.525014 31528 log.cc:824] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:40.526959 31528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5: No bootstrap required, opened a new log
I20260925 02:56:40.527981 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.516161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.12.222.254:45887)
0925 02:56:40.516593 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:40.516626 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:40.516663 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:56:40.517300 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:40.517318 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:40.517610 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:40.517983 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:40.518763 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.519671 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.520534 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.520978 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.521140 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.523592 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:40.523596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:40.523607 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:40.523633 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:40.525341 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:40.525782 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:40.525787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:40.525790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:40.525900 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:40.526190 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:40.526192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:40.526193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:40.526519 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:40.526651 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:40.526962 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 02:56:40.527340 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":140,"threads_started":1}
I20260925 02:56:40.530696 31528 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f7eb01f96d4173b58d70b57ba728c5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 45887 } }
I20260925 02:56:40.530902 31528 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:40.530975 31528 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f7eb01f96d4173b58d70b57ba728c5, State: Initialized, Role: FOLLOWER
I20260925 02:56:40.531352 31528 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f7eb01f96d4173b58d70b57ba728c5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 45887 } }
I20260925 02:56:40.531520 31528 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:40.531587 31528 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:40.531715 31528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:40.532604 31528 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f7eb01f96d4173b58d70b57ba728c5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 45887 } }
I20260925 02:56:40.532912 31528 leader_election.cc:302] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0f7eb01f96d4173b58d70b57ba728c5; no voters:
I20260925 02:56:40.533207 31528 leader_election.cc:288] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:40.533377 31533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:40.533720 31533 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5 [term 1 LEADER]: Becoming Leader. State: Replica: c0f7eb01f96d4173b58d70b57ba728c5, State: Running, Role: LEADER
I20260925 02:56:40.533998 31533 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f7eb01f96d4173b58d70b57ba728c5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 45887 } }
I20260925 02:56:40.534334 31528 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:40.535598 31535 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0f7eb01f96d4173b58d70b57ba728c5. Latest consensus state: current_term: 1 leader_uuid: "c0f7eb01f96d4173b58d70b57ba728c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f7eb01f96d4173b58d70b57ba728c5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 45887 } } }
I20260925 02:56:40.535645 31534 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0f7eb01f96d4173b58d70b57ba728c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f7eb01f96d4173b58d70b57ba728c5" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 45887 } } }
I20260925 02:56:40.535714 31535 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:40.535737 31534 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:40.536077 31542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:40.538463 31542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:40.543721 31542 catalog_manager.cc:1347] Generated new cluster ID: bf3c258b19874e6695337b2b4569e130
I20260925 02:56:40.543781 31542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:40.566139 31542 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:40.567170 31542 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:40.579638 31542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5: Generated new TSK 0
I20260925 02:56:40.580210 31542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
I20260925 02:56:40.615589 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:45887
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:40.727159 31554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:40.727351 31554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:40.727386 31554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:40.730753 31554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:40.730793 31554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:40.730858 31554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:56:40.735596 31554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:45887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:40.736346 31554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:40.737176 31554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:40.745553 31560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:40.745612 31554 server_base.cc:1034] running on GCE node
W20260925 02:56:40.745570 31559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:40.745568 31562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:40.746243 31554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:40.746788 31554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:40.747960 31554 hybrid_clock.cc:648] HybridClock initialized: now 1790305000747937 us; error 35 us; skew 500 ppm
Sep 25 02:56:40 dist-test-slave-252r krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305000, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:40.750488 31554 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:56:40.751250 31554 webserver.cc:460] Webserver started at http://127.12.222.193:42233/ using document root <none> and password file <none>
I20260925 02:56:40.751701 31554 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:40.751754 31554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:40.751910 31554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:40.753598 31554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "eae8db5bd8854c20ab00795ecbe8451f"
format_stamp: "Formatted at 2026-09-25 02:56:40 on dist-test-slave-252r"
server_key: "afbc7dcd300c900a925181e7ee4f3d8d"
server_key_iv: "89e1d99b1c30ec468e8124de5d2aed67"
server_key_version: "encryptionkey@0"
I20260925 02:56:40.754060 31554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "eae8db5bd8854c20ab00795ecbe8451f"
format_stamp: "Formatted at 2026-09-25 02:56:40 on dist-test-slave-252r"
server_key: "afbc7dcd300c900a925181e7ee4f3d8d"
server_key_iv: "89e1d99b1c30ec468e8124de5d2aed67"
server_key_version: "encryptionkey@0"
I20260925 02:56:40.757112 31554 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:56:40.759262 31569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:40.760347 31554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:56:40.760432 31554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "eae8db5bd8854c20ab00795ecbe8451f"
format_stamp: "Formatted at 2026-09-25 02:56:40 on dist-test-slave-252r"
server_key: "afbc7dcd300c900a925181e7ee4f3d8d"
server_key_iv: "89e1d99b1c30ec468e8124de5d2aed67"
server_key_version: "encryptionkey@0"
I20260925 02:56:40.760501 31554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:40.769551 31554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:40.771780 31554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:40.771900 31554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:40.772356 31554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:40.772984 31554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:40.773025 31554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:40.773056 31554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:40.773077 31554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:40.781533 31554 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:36501
I20260925 02:56:40.781564 31683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:36501 every 8 connection(s)
I20260925 02:56:40.782336 31554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
Sep 25 02:56:40 dist-test-slave-252r krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305000, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:56:40.790840 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 31554
I20260925 02:56:40.790975 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:56:40.791270 13179 external_mini_cluster.cc:1428] Setting key 859657e71a26ba20b87babcdc46517a7
I20260925 02:56:40.795059 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.783895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:53703 (local address 127.12.222.254:45887)
0925 02:56:40.784039 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:40.784044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:40.784949 (+ 905us) server_negotiation.cc:407] Connection header received
0925 02:56:40.786156 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:40.786158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:40.786200 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:40.786261 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:40.787234 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.787740 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.788843 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.789062 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.789139 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.791522 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:40.791526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:40.791529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:40.791553 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:40.793032 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:40.793550 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:40.793553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:40.793555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:40.793635 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:40.793978 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:40.793981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:40.793983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:40.794173 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:40.794281 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:40.794752 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 02:56:40.794878 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:56:40.795763 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.784337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:45887 (local address 127.12.222.193:53703)
0925 02:56:40.784785 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:40.784823 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:40.785939 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:40.786401 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:40.786407 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:40.786688 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:40.787116 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:40.787128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.787835 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.787838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:40.788742 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:40.788751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.789182 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.789184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:40.789304 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.789844 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:40.789859 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:40.791309 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:40.793148 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:40.793153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:40.793164 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:40.793440 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:40.793726 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:40.793729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:40.793731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:40.793865 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:40.794280 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:40.794287 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:40.794515 (+ 228us) client_negotiation.cc:769] Sending connection context
0925 02:56:40.794706 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 02:56:40.795020 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":121,"threads_started":1}
I20260925 02:56:40.796957 31684 heartbeater.cc:347] Connected to a master server at 127.12.222.254:45887
I20260925 02:56:40.797190 31684 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:40.797750 31684 heartbeater.cc:511] Master 127.12.222.254:45887 requested a full tablet report, sending...
I20260925 02:56:40.799201 31491 ts_manager.cc:194] Registered new tserver with Master: eae8db5bd8854c20ab00795ecbe8451f (127.12.222.193:36501)
I20260925 02:56:40.800366 31491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:53703
WARNING: no policy specified for kudu/127.12.222.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
I20260925 02:56:40.816179 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:45887
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:40.915094 31689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:40.915293 31689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:40.915347 31689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:40.919049 31689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:40.919092 31689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:40.919138 31689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:56:40.923753 31689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:45887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:40.924657 31689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:40.925480 31689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:40.933655 31695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:40.933785 31689 server_base.cc:1034] running on GCE node
W20260925 02:56:40.933655 31694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:40.933684 31697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:40.934291 31689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:40.934959 31689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:40.936129 31689 hybrid_clock.cc:648] HybridClock initialized: now 1790305000936105 us; error 35 us; skew 500 ppm
Sep 25 02:56:40 dist-test-slave-252r krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305000, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:40.942077 31689 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:56:40.942814 31689 webserver.cc:460] Webserver started at http://127.12.222.194:42737/ using document root <none> and password file <none>
I20260925 02:56:40.943301 31689 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:40.943375 31689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:40.943610 31689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:40.945441 31689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "6e162c3745c04dd18044a7a42fb7107c"
format_stamp: "Formatted at 2026-09-25 02:56:40 on dist-test-slave-252r"
server_key: "f8be785bc9a3f8487729aea8aafcbb13"
server_key_iv: "f1c837d6d300259fa06cd8d730b8f924"
server_key_version: "encryptionkey@0"
I20260925 02:56:40.945935 31689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "6e162c3745c04dd18044a7a42fb7107c"
format_stamp: "Formatted at 2026-09-25 02:56:40 on dist-test-slave-252r"
server_key: "f8be785bc9a3f8487729aea8aafcbb13"
server_key_iv: "f1c837d6d300259fa06cd8d730b8f924"
server_key_version: "encryptionkey@0"
I20260925 02:56:40.949075 31689 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:56:40.951200 31704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:40.952363 31689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:56:40.952472 31689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "6e162c3745c04dd18044a7a42fb7107c"
format_stamp: "Formatted at 2026-09-25 02:56:40 on dist-test-slave-252r"
server_key: "f8be785bc9a3f8487729aea8aafcbb13"
server_key_iv: "f1c837d6d300259fa06cd8d730b8f924"
server_key_version: "encryptionkey@0"
I20260925 02:56:40.952558 31689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:40.966357 31689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:40.969236 31689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:40.969410 31689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:40.969926 31689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:40.970671 31689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:40.970716 31689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:40.970777 31689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:40.970813 31689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:40.978236 31689 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:36669
I20260925 02:56:40.978305 31819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:36669 every 8 connection(s)
I20260925 02:56:40.979089 31689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:56:40.980866 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 31689
I20260925 02:56:40.981004 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:56:40.981274 13179 external_mini_cluster.cc:1428] Setting key d2945271e389d2625d03848280d69139
Sep 25 02:56:40 dist-test-slave-252r krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305000, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.222.195@KRBTEST.COM; defaulting to no policy
I20260925 02:56:40.992641 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.980660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:42233 (local address 127.12.222.254:45887)
0925 02:56:40.980835 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:40.980840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:40.981647 (+ 807us) server_negotiation.cc:407] Connection header received
0925 02:56:40.983084 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:40.983087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:40.983139 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:40.983230 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:40.984211 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.984891 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.986608 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.986806 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.986911 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.989331 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:40.989334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:40.989336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:40.989356 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:40.990705 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:40.991229 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:40.991232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:40.991233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:40.991316 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:40.991632 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:40.991636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:40.991637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:40.991793 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:40.991904 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:40.992364 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 02:56:40.992491 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.195@KRBTEST.COM" created.
I20260925 02:56:40.993309 31822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:40.981031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:45887 (local address 127.12.222.194:42233)
0925 02:56:40.981497 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:40.981547 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:40.982901 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:40.983351 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:40.983357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:40.983650 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:40.984039 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:40.984050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.985025 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.985030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:40.986487 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:40.986500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:40.986971 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:40.986977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:40.987189 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:40.987688 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:40.987709 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:40.989177 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:40.990826 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:40.990831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:40.990843 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:40.991105 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:40.991406 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:40.991409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:40.991410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:40.991540 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:40.991895 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:40.991900 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:40.992118 (+ 218us) client_negotiation.cc:769] Sending connection context
0925 02:56:40.992319 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 02:56:40.992599 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":98,"threads_started":1}
I20260925 02:56:40.994386 31820 heartbeater.cc:347] Connected to a master server at 127.12.222.254:45887
I20260925 02:56:40.994628 31820 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:40.995042 31820 heartbeater.cc:511] Master 127.12.222.254:45887 requested a full tablet report, sending...
I20260925 02:56:40.995999 31491 ts_manager.cc:194] Registered new tserver with Master: 6e162c3745c04dd18044a7a42fb7107c (127.12.222.194:36669)
I20260925 02:56:40.996573 31491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:42233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
I20260925 02:56:41.006842 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:45887
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:41.108829 31826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:41.109015 31826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:41.109050 31826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:41.112483 31826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:41.112524 31826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:41.112574 31826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:56:41.117103 31826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:45887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:41.117795 31826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:41.118564 31826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:41.126848 31831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:41.126883 31834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:41.126856 31832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:41.127499 31826 server_base.cc:1034] running on GCE node
I20260925 02:56:41.127851 31826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:41.128369 31826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:41.129540 31826 hybrid_clock.cc:648] HybridClock initialized: now 1790305001129525 us; error 26 us; skew 500 ppm
Sep 25 02:56:41 dist-test-slave-252r krb5kdc[31446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305001, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:41.131981 31826 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:56:41.132761 31826 webserver.cc:460] Webserver started at http://127.12.222.195:44657/ using document root <none> and password file <none>
I20260925 02:56:41.133224 31826 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:41.133306 31826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:41.133535 31826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:41.135479 31826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "926e718a2c4741648e215b1e2fa4d4e7"
format_stamp: "Formatted at 2026-09-25 02:56:41 on dist-test-slave-252r"
server_key: "abeab1ae49cc42688e8ede45401e101a"
server_key_iv: "7984c9d75230b4394dcf8e5a6a87ce46"
server_key_version: "encryptionkey@0"
I20260925 02:56:41.136018 31826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "926e718a2c4741648e215b1e2fa4d4e7"
format_stamp: "Formatted at 2026-09-25 02:56:41 on dist-test-slave-252r"
server_key: "abeab1ae49cc42688e8ede45401e101a"
server_key_iv: "7984c9d75230b4394dcf8e5a6a87ce46"
server_key_version: "encryptionkey@0"
I20260925 02:56:41.139125 31826 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:56:41.141149 31841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:41.142164 31826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:56:41.142285 31826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "926e718a2c4741648e215b1e2fa4d4e7"
format_stamp: "Formatted at 2026-09-25 02:56:41 on dist-test-slave-252r"
server_key: "abeab1ae49cc42688e8ede45401e101a"
server_key_iv: "7984c9d75230b4394dcf8e5a6a87ce46"
server_key_version: "encryptionkey@0"
I20260925 02:56:41.142380 31826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:41.155016 31826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:41.157526 31826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:41.157660 31826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:41.158123 31826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:41.158774 31826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:41.158818 31826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:41.158851 31826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:41.158871 31826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:41.167199 31826 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:45533
I20260925 02:56:41.167223 31957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:45533 every 8 connection(s)
I20260925 02:56:41.167910 31826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:56:41.171905 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 31826
I20260925 02:56:41.172029 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:56:41.172361 13179 external_mini_cluster.cc:1428] Setting key 81c09b8463e66842a4a4f46f6a343a30
Sep 25 02:56:41 dist-test-slave-252r krb5kdc[31446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305001, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:56:41.180094 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:41.169427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:47291 (local address 127.12.222.254:45887)
0925 02:56:41.169554 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:41.169557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:41.170224 (+ 667us) server_negotiation.cc:407] Connection header received
0925 02:56:41.171422 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:41.171426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:41.171480 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:41.171577 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:41.172755 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.173271 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.174749 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.174953 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.175032 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:41.177109 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:41.177112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:41.177114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:41.177137 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:41.178346 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:41.178777 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:41.178779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:41.178781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:41.178848 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:41.179125 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:41.179127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:41.179128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:41.179384 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:41.179479 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:41.179900 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 02:56:41.179989 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:56:41.180719 31960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:41.169726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:45887 (local address 127.12.222.195:47291)
0925 02:56:41.170096 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:41.170131 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:41.171225 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:41.171723 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:41.171729 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:41.172083 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:41.172595 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:41.172609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.173418 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.173421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:41.174650 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:41.174660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.175072 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.175075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:41.175195 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:41.175686 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:41.175701 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:41.176970 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:41.178462 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:41.178465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:41.178475 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:41.178696 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:41.178931 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:41.178933 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:41.178934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:41.179050 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:41.179487 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:41.179499 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:41.179685 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 02:56:41.179842 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 02:56:41.180035 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":96,"threads_started":1}
I20260925 02:56:41.181771 31958 heartbeater.cc:347] Connected to a master server at 127.12.222.254:45887
I20260925 02:56:41.181990 31958 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:41.182397 31958 heartbeater.cc:511] Master 127.12.222.254:45887 requested a full tablet report, sending...
I20260925 02:56:41.183292 31491 ts_manager.cc:194] Registered new tserver with Master: 926e718a2c4741648e215b1e2fa4d4e7 (127.12.222.195:45533)
I20260925 02:56:41.183831 31491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:47291
I20260925 02:56:41.186085 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:56:41.194299 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:41.187832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.12.222.254:45887)
0925 02:56:41.187943 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:41.187945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:41.188137 (+ 192us) server_negotiation.cc:407] Connection header received
0925 02:56:41.188215 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:41.188217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:41.188296 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:41.188354 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:41.188889 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.189302 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.190054 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.190242 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.190311 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:41.191668 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:41.191670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:41.191672 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:41.191690 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:41.192858 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:41.193282 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:41.193284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:41.193285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:41.193374 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:41.193647 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:41.193649 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:41.193650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:41.193818 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:41.193928 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:41.194136 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 02:56:41.194204 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:56:41.196928 31491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:56:41.199609 31491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:56:41.212780 31970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:41.207110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:45533 (local address 127.0.0.1:36258)
0925 02:56:41.207688 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:41.207735 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:41.207860 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:41.208513 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:41.208520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:41.208543 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:41.208673 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:41.208679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.209853 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.209857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:41.210976 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:41.210985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.212258 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.212262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:41.212371 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:41.212434 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 02:56:41.212536 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 02:56:41.212641 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":23,"spinlock_wait_cycles":6400,"thread_start_us":100,"threads_started":1}
I20260925 02:56:41.212821 31976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:41.207673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36258 (local address 127.12.222.195:45533)
0925 02:56:41.208025 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:41.208036 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:41.208055 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:56:41.208115 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:41.208120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:41.208345 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:41.208516 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:41.208809 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.209733 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.211297 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.212106 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.212260 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:41.212390 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:41.212586 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 02:56:41.212674 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":85,"threads_started":1}
I20260925 02:56:41.213021 31969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:41.207031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:36669 (local address 127.0.0.1:58504)
0925 02:56:41.207688 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:41.207734 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:41.207859 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:41.209223 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:41.209226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:41.209243 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:41.209331 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:41.209336 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.210545 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.210550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:41.211608 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:41.211617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.212639 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.212643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:41.212731 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:41.212752 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 02:56:41.212823 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 02:56:41.212906 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":4,"spinlock_wait_cycles":4864,"thread_start_us":71,"threads_started":1}
I20260925 02:56:41.213295 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:41.207371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:36501 (local address 127.0.0.1:57838)
0925 02:56:41.208289 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:41.208300 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:41.208399 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:41.208864 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:41.208866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:41.208891 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:41.208956 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:41.208962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.210307 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.210310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:41.211341 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:41.211351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.212602 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.212605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:41.212686 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:41.212708 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:56:41.212776 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 02:56:41.212997 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":80,"threads_started":1}
I20260925 02:56:41.213449 31971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:41.207144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58504 (local address 127.12.222.194:36669)
0925 02:56:41.208801 (+ 1657us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:41.208813 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:41.208835 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:56:41.208912 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:41.208916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:41.209083 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:41.209230 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:41.209450 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.210431 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.211727 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.212341 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.212454 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:41.212579 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:41.213204 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 02:56:41.213301 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1559,"thread_start_us":131,"threads_started":1}
I20260925 02:56:41.214157 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:41.207618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57838 (local address 127.12.222.193:36501)
0925 02:56:41.208147 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:41.208159 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:41.208385 (+ 226us) server_negotiation.cc:407] Connection header received
0925 02:56:41.208549 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:41.208554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:41.208747 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:41.208884 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:41.209078 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.210169 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.211485 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.212284 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.212401 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:41.212537 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:41.213162 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 02:56:41.213265 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":133,"threads_started":1}
I20260925 02:56:41.215838 31890 tablet_service.cc:1467] Processing CreateTablet for tablet 446eb32caa0848bca57ca38d160ff46f (DEFAULT_TABLE table=test-table [id=06c0ca4e33df43038622fd39eb842642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:41.215838 31752 tablet_service.cc:1467] Processing CreateTablet for tablet 446eb32caa0848bca57ca38d160ff46f (DEFAULT_TABLE table=test-table [id=06c0ca4e33df43038622fd39eb842642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:41.216336 31617 tablet_service.cc:1467] Processing CreateTablet for tablet 446eb32caa0848bca57ca38d160ff46f (DEFAULT_TABLE table=test-table [id=06c0ca4e33df43038622fd39eb842642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:41.216905 31752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 446eb32caa0848bca57ca38d160ff46f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:41.216905 31890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 446eb32caa0848bca57ca38d160ff46f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:41.217232 31617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 446eb32caa0848bca57ca38d160ff46f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:41.223829 31979 tablet_bootstrap.cc:492] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c: Bootstrap starting.
I20260925 02:56:41.224026 31977 tablet_bootstrap.cc:492] T 446eb32caa0848bca57ca38d160ff46f P eae8db5bd8854c20ab00795ecbe8451f: Bootstrap starting.
I20260925 02:56:41.224531 31978 tablet_bootstrap.cc:492] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7: Bootstrap starting.
I20260925 02:56:41.226085 31979 tablet_bootstrap.cc:654] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:41.226085 31977 tablet_bootstrap.cc:654] T 446eb32caa0848bca57ca38d160ff46f P eae8db5bd8854c20ab00795ecbe8451f: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:41.226115 31978 tablet_bootstrap.cc:654] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:41.227098 31978 log.cc:824] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:41.227341 31979 log.cc:824] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:41.227341 31977 log.cc:824] T 446eb32caa0848bca57ca38d160ff46f P eae8db5bd8854c20ab00795ecbe8451f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:41.229164 31978 tablet_bootstrap.cc:492] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7: No bootstrap required, opened a new log
I20260925 02:56:41.229262 31977 tablet_bootstrap.cc:492] T 446eb32caa0848bca57ca38d160ff46f P eae8db5bd8854c20ab00795ecbe8451f: No bootstrap required, opened a new log
I20260925 02:56:41.229346 31979 tablet_bootstrap.cc:492] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c: No bootstrap required, opened a new log
I20260925 02:56:41.229359 31978 ts_tablet_manager.cc:1397] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:56:41.229447 31977 ts_tablet_manager.cc:1397] T 446eb32caa0848bca57ca38d160ff46f P eae8db5bd8854c20ab00795ecbe8451f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:56:41.229542 31979 ts_tablet_manager.cc:1397] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 02:56:41.232115 31978 raft_consensus.cc:348] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e162c3745c04dd18044a7a42fb7107c" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36669 } } peers { permanent_uuid: "926e718a2c4741648e215b1e2fa4d4e7" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45533 } } peers { permanent_uuid: "eae8db5bd8854c20ab00795ecbe8451f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 36501 } }
I20260925 02:56:41.232373 31978 raft_consensus.cc:374] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:41.232425 31978 raft_consensus.cc:729] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926e718a2c4741648e215b1e2fa4d4e7, State: Initialized, Role: FOLLOWER
I20260925 02:56:41.232824 31978 consensus_queue.cc:260] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e162c3745c04dd18044a7a42fb7107c" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36669 } } peers { permanent_uuid: "926e718a2c4741648e215b1e2fa4d4e7" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45533 } } peers { permanent_uuid: "eae8db5bd8854c20ab00795ecbe8451f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 36501 } }
I20260925 02:56:41.233115 31977 raft_consensus.cc:348] T 446eb32caa0848bca57ca38d160ff46f P eae8db5bd8854c20ab00795ecbe8451f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e162c3745c04dd18044a7a42fb7107c" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36669 } } peers { permanent_uuid: "926e718a2c4741648e215b1e2fa4d4e7" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45533 } } peers { permanent_uuid: "eae8db5bd8854c20ab00795ecbe8451f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 36501 } }
I20260925 02:56:41.233115 31979 raft_consensus.cc:348] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e162c3745c04dd18044a7a42fb7107c" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36669 } } peers { permanent_uuid: "926e718a2c4741648e215b1e2fa4d4e7" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45533 } } peers { permanent_uuid: "eae8db5bd8854c20ab00795ecbe8451f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 36501 } }
I20260925 02:56:41.233340 31977 raft_consensus.cc:374] T 446eb32caa0848bca57ca38d160ff46f P eae8db5bd8854c20ab00795ecbe8451f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:41.233340 31979 raft_consensus.cc:374] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:41.233395 31958 heartbeater.cc:503] Master 127.12.222.254:45887 was elected leader, sending a full tablet report...
I20260925 02:56:41.233430 31979 raft_consensus.cc:729] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e162c3745c04dd18044a7a42fb7107c, State: Initialized, Role: FOLLOWER
I20260925 02:56:41.233417 31977 raft_consensus.cc:729] T 446eb32caa0848bca57ca38d160ff46f P eae8db5bd8854c20ab00795ecbe8451f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae8db5bd8854c20ab00795ecbe8451f, State: Initialized, Role: FOLLOWER
I20260925 02:56:41.233755 31978 ts_tablet_manager.cc:1428] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260925 02:56:41.233878 31979 consensus_queue.cc:260] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e162c3745c04dd18044a7a42fb7107c" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36669 } } peers { permanent_uuid: "926e718a2c4741648e215b1e2fa4d4e7" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45533 } } peers { permanent_uuid: "eae8db5bd8854c20ab00795ecbe8451f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 36501 } }
I20260925 02:56:41.233878 31977 consensus_queue.cc:260] T 446eb32caa0848bca57ca38d160ff46f P eae8db5bd8854c20ab00795ecbe8451f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e162c3745c04dd18044a7a42fb7107c" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36669 } } peers { permanent_uuid: "926e718a2c4741648e215b1e2fa4d4e7" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45533 } } peers { permanent_uuid: "eae8db5bd8854c20ab00795ecbe8451f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 36501 } }
I20260925 02:56:41.234771 31820 heartbeater.cc:503] Master 127.12.222.254:45887 was elected leader, sending a full tablet report...
I20260925 02:56:41.234784 31977 ts_tablet_manager.cc:1428] T 446eb32caa0848bca57ca38d160ff46f P eae8db5bd8854c20ab00795ecbe8451f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 02:56:41.234843 31979 ts_tablet_manager.cc:1428] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 02:56:41.235590 31684 heartbeater.cc:503] Master 127.12.222.254:45887 was elected leader, sending a full tablet report...
W20260925 02:56:41.283376 31685 tablet.cc:2369] T 446eb32caa0848bca57ca38d160ff46f P eae8db5bd8854c20ab00795ecbe8451f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:41.411504 31985 raft_consensus.cc:482] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:56:41.411723 31985 raft_consensus.cc:504] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e162c3745c04dd18044a7a42fb7107c" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36669 } } peers { permanent_uuid: "926e718a2c4741648e215b1e2fa4d4e7" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45533 } } peers { permanent_uuid: "eae8db5bd8854c20ab00795ecbe8451f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 36501 } }
I20260925 02:56:41.412882 31985 leader_election.cc:288] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 926e718a2c4741648e215b1e2fa4d4e7 (127.12.222.195:45533), eae8db5bd8854c20ab00795ecbe8451f (127.12.222.193:36501)
I20260925 02:56:41.417857 31976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:41.413199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:57027 (local address 127.12.222.195:45533)
0925 02:56:41.413321 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:41.413327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:41.413345 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:56:41.413516 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:41.413519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:41.413598 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:41.413682 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:41.414002 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.414826 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.416580 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.417152 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.417273 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:41.417322 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:41.417671 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 02:56:41.417734 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:56:41.417857 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:41.413278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:37153 (local address 127.12.222.193:36501)
0925 02:56:41.413456 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:41.413461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:41.413612 (+ 151us) server_negotiation.cc:407] Connection header received
0925 02:56:41.413691 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:41.413693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:41.413741 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:41.413811 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:41.414069 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.414826 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.416580 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.417152 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.417274 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:41.417322 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:41.417671 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 02:56:41.417734 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:56:41.417888 31986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:41.413091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:36501 (local address 127.12.222.194:37153)
0925 02:56:41.413507 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:41.413520 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:41.413612 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:41.413835 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:41.413838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:41.413852 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:41.413955 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:41.413962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.414940 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.414943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:41.416433 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:41.416444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.417273 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.417277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:41.417483 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:41.417562 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 02:56:41.417641 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 02:56:41.417732 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":232,"thread_start_us":116,"threads_started":1}
I20260925 02:56:41.417887 31822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:41.413071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:45533 (local address 127.12.222.194:57027)
0925 02:56:41.413248 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:41.413265 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:41.413359 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:41.413720 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:41.413722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:41.413734 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:41.413871 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:41.413886 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.414940 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.414943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:41.416438 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:41.416444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.417273 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.417276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:41.417511 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:41.417562 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 02:56:41.417641 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 02:56:41.417732 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":271}
I20260925 02:56:41.418553 31638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "446eb32caa0848bca57ca38d160ff46f" candidate_uuid: "6e162c3745c04dd18044a7a42fb7107c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eae8db5bd8854c20ab00795ecbe8451f" is_pre_election: true
W20260925 02:56:41.418848 31959 tablet.cc:2369] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:41.418915 31638 raft_consensus.cc:2454] T 446eb32caa0848bca57ca38d160ff46f P eae8db5bd8854c20ab00795ecbe8451f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e162c3745c04dd18044a7a42fb7107c in term 0.
I20260925 02:56:41.418995 31912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "446eb32caa0848bca57ca38d160ff46f" candidate_uuid: "6e162c3745c04dd18044a7a42fb7107c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926e718a2c4741648e215b1e2fa4d4e7" is_pre_election: true
I20260925 02:56:41.419266 31912 raft_consensus.cc:2454] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e162c3745c04dd18044a7a42fb7107c in term 0.
I20260925 02:56:41.419386 31708 leader_election.cc:302] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e162c3745c04dd18044a7a42fb7107c, eae8db5bd8854c20ab00795ecbe8451f; no voters:
I20260925 02:56:41.419646 31985 raft_consensus.cc:2781] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:56:41.419749 31985 raft_consensus.cc:482] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:56:41.419803 31985 raft_consensus.cc:3037] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:41.420904 31985 raft_consensus.cc:504] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e162c3745c04dd18044a7a42fb7107c" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36669 } } peers { permanent_uuid: "926e718a2c4741648e215b1e2fa4d4e7" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45533 } } peers { permanent_uuid: "eae8db5bd8854c20ab00795ecbe8451f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 36501 } }
I20260925 02:56:41.421219 31985 leader_election.cc:288] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [CANDIDATE]: Term 1 election: Requested vote from peers 926e718a2c4741648e215b1e2fa4d4e7 (127.12.222.195:45533), eae8db5bd8854c20ab00795ecbe8451f (127.12.222.193:36501)
I20260925 02:56:41.421525 31912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "446eb32caa0848bca57ca38d160ff46f" candidate_uuid: "6e162c3745c04dd18044a7a42fb7107c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926e718a2c4741648e215b1e2fa4d4e7"
I20260925 02:56:41.421630 31638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "446eb32caa0848bca57ca38d160ff46f" candidate_uuid: "6e162c3745c04dd18044a7a42fb7107c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eae8db5bd8854c20ab00795ecbe8451f"
I20260925 02:56:41.421636 31912 raft_consensus.cc:3037] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:41.421751 31638 raft_consensus.cc:3037] T 446eb32caa0848bca57ca38d160ff46f P eae8db5bd8854c20ab00795ecbe8451f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:41.422657 31912 raft_consensus.cc:2454] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e162c3745c04dd18044a7a42fb7107c in term 1.
I20260925 02:56:41.422744 31638 raft_consensus.cc:2454] T 446eb32caa0848bca57ca38d160ff46f P eae8db5bd8854c20ab00795ecbe8451f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e162c3745c04dd18044a7a42fb7107c in term 1.
I20260925 02:56:41.422956 31707 leader_election.cc:302] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e162c3745c04dd18044a7a42fb7107c, 926e718a2c4741648e215b1e2fa4d4e7; no voters:
I20260925 02:56:41.423120 31985 raft_consensus.cc:2781] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:41.423301 31985 raft_consensus.cc:686] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [term 1 LEADER]: Becoming Leader. State: Replica: 6e162c3745c04dd18044a7a42fb7107c, State: Running, Role: LEADER
I20260925 02:56:41.423623 31985 consensus_queue.cc:237] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e162c3745c04dd18044a7a42fb7107c" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36669 } } peers { permanent_uuid: "926e718a2c4741648e215b1e2fa4d4e7" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45533 } } peers { permanent_uuid: "eae8db5bd8854c20ab00795ecbe8451f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 36501 } }
I20260925 02:56:41.426520 31489 catalog_manager.cc:5495] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e162c3745c04dd18044a7a42fb7107c (127.12.222.194). New cstate: current_term: 1 leader_uuid: "6e162c3745c04dd18044a7a42fb7107c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e162c3745c04dd18044a7a42fb7107c" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 36669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "926e718a2c4741648e215b1e2fa4d4e7" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eae8db5bd8854c20ab00795ecbe8451f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 36501 } health_report { overall_health: UNKNOWN } } }
I20260925 02:56:41.444348 31971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:41.440783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58514 (local address 127.12.222.194:36669)
0925 02:56:41.440919 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:41.440925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:41.440937 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:56:41.441111 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:41.441113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:41.441157 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:41.441250 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:41.441503 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.441983 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.442835 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.443049 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.443153 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:41.443515 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:41.443871 (+ 356us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:41.444029 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:41.444120 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 02:56:41.444157 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:56:41.449441 31912 raft_consensus.cc:1261] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7 [term 1 FOLLOWER]: Refusing update from remote peer 6e162c3745c04dd18044a7a42fb7107c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:41.449568 31638 raft_consensus.cc:1261] T 446eb32caa0848bca57ca38d160ff46f P eae8db5bd8854c20ab00795ecbe8451f [term 1 FOLLOWER]: Refusing update from remote peer 6e162c3745c04dd18044a7a42fb7107c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:41.450071 31985 consensus_queue.cc:1035] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [LEADER]: Connected to new peer: Peer: permanent_uuid: "926e718a2c4741648e215b1e2fa4d4e7" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:41.450249 31987 consensus_queue.cc:1035] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae8db5bd8854c20ab00795ecbe8451f" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 36501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:41.465370 31489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:41.465499 31489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:41.467706 31489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0f7eb01f96d4173b58d70b57ba728c5: Sending DeleteTablet for 3 replicas of tablet 446eb32caa0848bca57ca38d160ff46f
I20260925 02:56:41.468413 31752 tablet_service.cc:1514] Processing DeleteTablet for tablet 446eb32caa0848bca57ca38d160ff46f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:41 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:58504
I20260925 02:56:41.468477 31890 tablet_service.cc:1514] Processing DeleteTablet for tablet 446eb32caa0848bca57ca38d160ff46f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:41 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:36258
I20260925 02:56:41.468879 31617 tablet_service.cc:1514] Processing DeleteTablet for tablet 446eb32caa0848bca57ca38d160ff46f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:41 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:57838
I20260925 02:56:41.469313 32002 tablet_replica.cc:330] T 446eb32caa0848bca57ca38d160ff46f P eae8db5bd8854c20ab00795ecbe8451f: stopping tablet replica
I20260925 02:56:41.469388 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 31554
I20260925 02:56:41.469533 32000 tablet_replica.cc:330] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c: stopping tablet replica
I20260925 02:56:41.469655 32002 raft_consensus.cc:2229] T 446eb32caa0848bca57ca38d160ff46f P eae8db5bd8854c20ab00795ecbe8451f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:41.469748 32000 raft_consensus.cc:2229] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:56:41.470059 32000 raft_consensus.cc:2258] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:41.469383 32001 tablet_replica.cc:330] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7: stopping tablet replica
I20260925 02:56:41.470458 32001 raft_consensus.cc:2229] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:41.470647 32001 raft_consensus.cc:2258] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:41.472195 32001 ts_tablet_manager.cc:1905] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:41.472195 32000 ts_tablet_manager.cc:1905] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:41.474789 32000 ts_tablet_manager.cc:1918] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:56:41.474789 32001 ts_tablet_manager.cc:1918] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:56:41.474880 32001 log.cc:1196] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/446eb32caa0848bca57ca38d160ff46f
I20260925 02:56:41.474975 32000 log.cc:1196] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/446eb32caa0848bca57ca38d160ff46f
I20260925 02:56:41.475291 32001 ts_tablet_manager.cc:1939] T 446eb32caa0848bca57ca38d160ff46f P 926e718a2c4741648e215b1e2fa4d4e7: Deleting consensus metadata
I20260925 02:56:41.475517 32000 ts_tablet_manager.cc:1939] T 446eb32caa0848bca57ca38d160ff46f P 6e162c3745c04dd18044a7a42fb7107c: Deleting consensus metadata
I20260925 02:56:41.476531 31477 catalog_manager.cc:4841] TS 926e718a2c4741648e215b1e2fa4d4e7 (127.12.222.195:45533): tablet 446eb32caa0848bca57ca38d160ff46f (table test-table [id=06c0ca4e33df43038622fd39eb842642]) successfully deleted
I20260925 02:56:41.476929 31475 catalog_manager.cc:4841] TS 6e162c3745c04dd18044a7a42fb7107c (127.12.222.194:36669): tablet 446eb32caa0848bca57ca38d160ff46f (table test-table [id=06c0ca4e33df43038622fd39eb842642]) successfully deleted
W20260925 02:56:41.481175 31478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.222.193:36501 (error 108)
I20260925 02:56:41.482129 31970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:41.481844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:36501 (local address 127.0.0.1:57854)
0925 02:56:41.481978 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:41.482049 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.193:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 02:56:41.482225 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 31689
W20260925 02:56:41.482368 31478 catalog_manager.cc:4584] TS eae8db5bd8854c20ab00795ecbe8451f (127.12.222.193:36501): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 446eb32caa0848bca57ca38d160ff46f: Network error: Client connection negotiation failed: client connection to 127.12.222.193:36501: connect: Connection refused (error 111)
I20260925 02:56:41.488494 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 31826
I20260925 02:56:41.497187 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 31459
2026-09-25T02:56:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2236 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T02:56:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:41Z Disabled control of system clock
I20260925 02:56:41.533950 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:40109
--webserver_interface=127.12.222.254
--webserver_port=0
--builtin_ntp_servers=127.12.222.212:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:40109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:56:41.653883 32007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:41.654098 32007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:41.654129 32007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:41.657660 32007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:41.657694 32007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:41.657707 32007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:41.657721 32007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:41.657733 32007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:41.661989 32007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:40109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:40109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:41.662799 32007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:41.663956 32007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:41.672938 32015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:41.673030 32007 server_base.cc:1034] running on GCE node
W20260925 02:56:41.672968 32013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:41.673154 32012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:41.673671 32007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:41.674685 32007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:41.675861 32007 hybrid_clock.cc:648] HybridClock initialized: now 1790305001675849 us; error 30 us; skew 500 ppm
I20260925 02:56:41.677615 32007 webserver.cc:460] Webserver started at http://127.12.222.254:39799/ using document root <none> and password file <none>
I20260925 02:56:41.678133 32007 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:41.678224 32007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:41.678440 32007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:41.680423 32007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "5f6d7badfd0f4c858518a3d6952f8c51"
format_stamp: "Formatted at 2026-09-25 02:56:41 on dist-test-slave-252r"
server_key: "9054283bff2dae8cadd22a3c02cc7540"
server_key_iv: "9e17cfc7a2e5b26eb7e93407d3f8c8aa"
server_key_version: "encryptionkey@0"
I20260925 02:56:41.681038 32007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "5f6d7badfd0f4c858518a3d6952f8c51"
format_stamp: "Formatted at 2026-09-25 02:56:41 on dist-test-slave-252r"
server_key: "9054283bff2dae8cadd22a3c02cc7540"
server_key_iv: "9e17cfc7a2e5b26eb7e93407d3f8c8aa"
server_key_version: "encryptionkey@0"
I20260925 02:56:41.684335 32007 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:56:41.686455 32021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:41.687502 32007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:41.687626 32007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "5f6d7badfd0f4c858518a3d6952f8c51"
format_stamp: "Formatted at 2026-09-25 02:56:41 on dist-test-slave-252r"
server_key: "9054283bff2dae8cadd22a3c02cc7540"
server_key_iv: "9e17cfc7a2e5b26eb7e93407d3f8c8aa"
server_key_version: "encryptionkey@0"
I20260925 02:56:41.687719 32007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:41.709620 32007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:41.710162 32007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:41.710361 32007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:41.716456 32007 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:40109
I20260925 02:56:41.716483 32074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:40109 every 8 connection(s)
I20260925 02:56:41.717226 32007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:41.719980 32075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:41.720674 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 32007
I20260925 02:56:41.720811 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:56:41.721128 13179 external_mini_cluster.cc:1428] Setting key ba7e0211d50784a687f8001628e65f6a
I20260925 02:56:41.725603 32075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51: Bootstrap starting.
I20260925 02:56:41.727551 32075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:41.728178 32075 log.cc:824] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:41.729417 32078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:41.722503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42950 (local address 127.12.222.254:40109)
0925 02:56:41.723070 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:41.723083 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:41.723112 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:56:41.723714 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:41.723730 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:41.724087 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:41.724422 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:41.724827 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.725640 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.726457 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.726883 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.727052 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:41.727884 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:41.727889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:41.727899 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:41.728223 (+ 324us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:41.728331 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:41.728356 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:41.728459 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:41.728565 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 02:56:41.728790 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":128,"threads_started":1}
I20260925 02:56:41.729928 32075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51: No bootstrap required, opened a new log
I20260925 02:56:41.733320 32075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f6d7badfd0f4c858518a3d6952f8c51" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 40109 } }
I20260925 02:56:41.733560 32075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:41.733640 32075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f6d7badfd0f4c858518a3d6952f8c51, State: Initialized, Role: FOLLOWER
I20260925 02:56:41.734076 32075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f6d7badfd0f4c858518a3d6952f8c51" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 40109 } }
I20260925 02:56:41.734248 32075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:41.734316 32075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:41.734412 32075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:41.735167 32075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f6d7badfd0f4c858518a3d6952f8c51" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 40109 } }
I20260925 02:56:41.735471 32075 leader_election.cc:302] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f6d7badfd0f4c858518a3d6952f8c51; no voters:
I20260925 02:56:41.735731 32075 leader_election.cc:288] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:41.735843 32080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:41.736060 32080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51 [term 1 LEADER]: Becoming Leader. State: Replica: 5f6d7badfd0f4c858518a3d6952f8c51, State: Running, Role: LEADER
I20260925 02:56:41.736334 32080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f6d7badfd0f4c858518a3d6952f8c51" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 40109 } }
I20260925 02:56:41.736819 32075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:41.737633 32082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f6d7badfd0f4c858518a3d6952f8c51. Latest consensus state: current_term: 1 leader_uuid: "5f6d7badfd0f4c858518a3d6952f8c51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f6d7badfd0f4c858518a3d6952f8c51" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 40109 } } }
I20260925 02:56:41.737756 32082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:41.737823 32081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f6d7badfd0f4c858518a3d6952f8c51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f6d7badfd0f4c858518a3d6952f8c51" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 40109 } } }
I20260925 02:56:41.737933 32081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:41.738193 32089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:41.740980 32089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:41.746114 32089 catalog_manager.cc:1347] Generated new cluster ID: ca3a8009a14f448bb91c57c60cbad65d
I20260925 02:56:41.746165 32089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:41.765172 32089 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:41.766204 32089 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:41.776206 32089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51: Generated new TSK 0
I20260925 02:56:41.776763 32089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:41.782128 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:40109
--builtin_ntp_servers=127.12.222.212:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:56:41.894275 32099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:41.894497 32099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:41.894532 32099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:41.898329 32099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:41.898373 32099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:41.898422 32099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:56:41.902782 32099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:40109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:41.903637 32099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:41.904959 32099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:41.913132 32105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:41.913161 32106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:41.913182 32099 server_base.cc:1034] running on GCE node
W20260925 02:56:41.913137 32108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:41.913671 32099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:41.914238 32099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:41.915426 32099 hybrid_clock.cc:648] HybridClock initialized: now 1790305001915399 us; error 40 us; skew 500 ppm
I20260925 02:56:41.917071 32099 webserver.cc:460] Webserver started at http://127.12.222.193:39955/ using document root <none> and password file <none>
I20260925 02:56:41.917630 32099 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:41.917722 32099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:41.917943 32099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:41.919818 32099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "2019fdbda6db44c6989f108f9d81c8b1"
format_stamp: "Formatted at 2026-09-25 02:56:41 on dist-test-slave-252r"
server_key: "5cca2fe48aebff0adba22c9e990018b9"
server_key_iv: "4d6595dfd8b386028b0198cfa1214fc2"
server_key_version: "encryptionkey@0"
I20260925 02:56:41.920403 32099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "2019fdbda6db44c6989f108f9d81c8b1"
format_stamp: "Formatted at 2026-09-25 02:56:41 on dist-test-slave-252r"
server_key: "5cca2fe48aebff0adba22c9e990018b9"
server_key_iv: "4d6595dfd8b386028b0198cfa1214fc2"
server_key_version: "encryptionkey@0"
I20260925 02:56:41.923804 32099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:56:41.926059 32114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:41.927171 32099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:41.927309 32099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "2019fdbda6db44c6989f108f9d81c8b1"
format_stamp: "Formatted at 2026-09-25 02:56:41 on dist-test-slave-252r"
server_key: "5cca2fe48aebff0adba22c9e990018b9"
server_key_iv: "4d6595dfd8b386028b0198cfa1214fc2"
server_key_version: "encryptionkey@0"
I20260925 02:56:41.927425 32099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:41.941588 32099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:41.942143 32099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:41.942310 32099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:41.942791 32099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:41.943604 32099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:41.943673 32099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:41.943740 32099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:41.943791 32099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:41.951813 32099 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:39699
I20260925 02:56:41.951838 32227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:39699 every 8 connection(s)
I20260925 02:56:41.952636 32099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:56:41.958799 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 32099
I20260925 02:56:41.958930 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:56:41.959189 13179 external_mini_cluster.cc:1428] Setting key 76e005cea0c1d520f18806b4b32a3293
I20260925 02:56:41.960867 32078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:41.954183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:40635 (local address 127.12.222.254:40109)
0925 02:56:41.954308 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:41.954312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:41.955145 (+ 833us) server_negotiation.cc:407] Connection header received
0925 02:56:41.956416 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:41.956420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:41.956484 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:41.956583 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:41.957326 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.957795 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.958778 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.959128 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.959228 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:41.960384 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:41.960387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:41.960389 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:41.960402 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:41.960420 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:41.960428 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:41.960505 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:41.960671 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 02:56:41.960737 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:56:41.961192 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:40109
--builtin_ntp_servers=127.12.222.212:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:56:41.961812 32230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:41.954455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:40109 (local address 127.12.222.193:40635)
0925 02:56:41.955014 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:41.955046 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:41.956169 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:41.956735 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:41.956742 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:41.956822 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:41.957194 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:41.957204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.957911 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.957915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:41.958663 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:41.958672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:41.959264 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:41.959267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:41.959447 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:41.960085 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:56:41.960086 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:41.960120 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:56:41.960173 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:41.960508 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:41.960520 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:41.960525 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:56:41.960660 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 02:56:41.960920 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":117,"threads_started":1}
I20260925 02:56:41.962958 32228 heartbeater.cc:347] Connected to a master server at 127.12.222.254:40109
I20260925 02:56:41.963163 32228 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:41.963611 32228 heartbeater.cc:511] Master 127.12.222.254:40109 requested a full tablet report, sending...
I20260925 02:56:41.964958 32038 ts_manager.cc:194] Registered new tserver with Master: 2019fdbda6db44c6989f108f9d81c8b1 (127.12.222.193:39699)
I20260925 02:56:41.965950 32038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.222.193:40635
W20260925 02:56:42.062064 32231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:42.062258 32231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:42.062289 32231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:42.065989 32231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:42.066027 32231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:42.066074 32231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:56:42.070106 32231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:40109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:42.070958 32231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:42.072103 32231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:42.080545 32236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:42.080552 32237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:42.080552 32239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:42.080794 32231 server_base.cc:1034] running on GCE node
I20260925 02:56:42.081112 32231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:42.081691 32231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:42.082886 32231 hybrid_clock.cc:648] HybridClock initialized: now 1790305002082866 us; error 35 us; skew 500 ppm
I20260925 02:56:42.084506 32231 webserver.cc:460] Webserver started at http://127.12.222.194:33765/ using document root <none> and password file <none>
I20260925 02:56:42.085049 32231 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:42.085144 32231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:42.085357 32231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:42.087333 32231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "6983bcbdaa0641e8923c39f9edb6cb07"
format_stamp: "Formatted at 2026-09-25 02:56:42 on dist-test-slave-252r"
server_key: "8c9b8e0a33073f1a3488196e5c6293d0"
server_key_iv: "abd84c2bc06c61dde208ce8d4a71656e"
server_key_version: "encryptionkey@0"
I20260925 02:56:42.087915 32231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "6983bcbdaa0641e8923c39f9edb6cb07"
format_stamp: "Formatted at 2026-09-25 02:56:42 on dist-test-slave-252r"
server_key: "8c9b8e0a33073f1a3488196e5c6293d0"
server_key_iv: "abd84c2bc06c61dde208ce8d4a71656e"
server_key_version: "encryptionkey@0"
I20260925 02:56:42.091236 32231 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 02:56:42.093426 32245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:42.094650 32231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:56:42.094776 32231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "6983bcbdaa0641e8923c39f9edb6cb07"
format_stamp: "Formatted at 2026-09-25 02:56:42 on dist-test-slave-252r"
server_key: "8c9b8e0a33073f1a3488196e5c6293d0"
server_key_iv: "abd84c2bc06c61dde208ce8d4a71656e"
server_key_version: "encryptionkey@0"
I20260925 02:56:42.094882 32231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:42.109331 32231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:42.109862 32231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:42.110044 32231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:42.110522 32231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:42.111284 32231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:42.111349 32231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:42.111410 32231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:42.111456 32231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:42.119524 32231 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:45609
I20260925 02:56:42.119549 32360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:45609 every 8 connection(s)
I20260925 02:56:42.120270 32231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:56:42.127709 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 32231
I20260925 02:56:42.127832 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:56:42.128123 13179 external_mini_cluster.cc:1428] Setting key a6b1a420192d15301ea233447648b9fa
I20260925 02:56:42.128422 32078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:42.121876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:36855 (local address 127.12.222.254:40109)
0925 02:56:42.121995 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:42.121999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:42.122841 (+ 842us) server_negotiation.cc:407] Connection header received
0925 02:56:42.124100 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:42.124104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:42.124164 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:42.124259 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:42.125047 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.125538 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.126589 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.126794 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.126892 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:42.127884 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:42.127888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:42.127890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:42.127903 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:42.127930 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:42.127938 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:42.128013 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:42.128179 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 02:56:42.128268 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:56:42.129205 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:42.122154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:40109 (local address 127.12.222.194:36855)
0925 02:56:42.122703 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:42.122736 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:42.123901 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:42.124413 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:42.124419 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:42.124494 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:42.124903 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:42.124914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.125663 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.125667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:42.126455 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:42.126463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.126936 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.126940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:42.127082 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:42.127628 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:56:42.127629 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:42.127663 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:56:42.127717 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:42.128028 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:42.128040 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:42.128048 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:56:42.128178 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 02:56:42.128450 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":99,"threads_started":1}
I20260925 02:56:42.130062 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:40109
--builtin_ntp_servers=127.12.222.212:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:56:42.130173 32361 heartbeater.cc:347] Connected to a master server at 127.12.222.254:40109
I20260925 02:56:42.130354 32361 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:42.130756 32361 heartbeater.cc:511] Master 127.12.222.254:40109 requested a full tablet report, sending...
I20260925 02:56:42.131645 32038 ts_manager.cc:194] Registered new tserver with Master: 6983bcbdaa0641e8923c39f9edb6cb07 (127.12.222.194:45609)
I20260925 02:56:42.132122 32038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.222.194:36855
W20260925 02:56:42.229399 32364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:42.229622 32364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:42.229681 32364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:42.233471 32364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:42.233534 32364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:42.233613 32364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:56:42.238389 32364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:40109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:42.239274 32364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:42.240545 32364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:42.249140 32370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:42.249130 32369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:42.249205 32364 server_base.cc:1034] running on GCE node
W20260925 02:56:42.249131 32372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:42.249621 32364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:42.250136 32364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:42.251281 32364 hybrid_clock.cc:648] HybridClock initialized: now 1790305002251263 us; error 27 us; skew 500 ppm
I20260925 02:56:42.252859 32364 webserver.cc:460] Webserver started at http://127.12.222.195:34869/ using document root <none> and password file <none>
I20260925 02:56:42.253341 32364 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:42.253403 32364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:42.253654 32364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:42.255427 32364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "025155ad33054ce7af52c5376cbecf16"
format_stamp: "Formatted at 2026-09-25 02:56:42 on dist-test-slave-252r"
server_key: "b19a3d093ca5e3a5eaade89873f65f28"
server_key_iv: "286bbd7e06d4886119a7f0441a9c01ca"
server_key_version: "encryptionkey@0"
I20260925 02:56:42.255924 32364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "025155ad33054ce7af52c5376cbecf16"
format_stamp: "Formatted at 2026-09-25 02:56:42 on dist-test-slave-252r"
server_key: "b19a3d093ca5e3a5eaade89873f65f28"
server_key_iv: "286bbd7e06d4886119a7f0441a9c01ca"
server_key_version: "encryptionkey@0"
I20260925 02:56:42.259176 32364 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 02:56:42.261452 32378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:42.262535 32364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:56:42.262660 32364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "025155ad33054ce7af52c5376cbecf16"
format_stamp: "Formatted at 2026-09-25 02:56:42 on dist-test-slave-252r"
server_key: "b19a3d093ca5e3a5eaade89873f65f28"
server_key_iv: "286bbd7e06d4886119a7f0441a9c01ca"
server_key_version: "encryptionkey@0"
I20260925 02:56:42.262753 32364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:42.271520 32364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:42.272013 32364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:42.272172 32364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:42.272684 32364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:42.273473 32364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:42.273517 32364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:42.273578 32364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:42.273615 32364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:42.281728 32364 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:46125
I20260925 02:56:42.281785 32493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:46125 every 8 connection(s)
I20260925 02:56:42.282521 32364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:56:42.285888 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 32364
I20260925 02:56:42.285971 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:56:42.286237 13179 external_mini_cluster.cc:1428] Setting key 9bb01723168fc98fc087c2b259dc7502
I20260925 02:56:42.290606 32078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:42.284072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:49079 (local address 127.12.222.254:40109)
0925 02:56:42.284215 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:42.284218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:42.285206 (+ 988us) server_negotiation.cc:407] Connection header received
0925 02:56:42.286549 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:42.286553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:42.286606 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:42.286699 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:42.287500 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.288027 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.289102 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.289296 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.289368 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:42.290223 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:42.290227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:42.290231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:42.290245 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:42.290269 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:42.290277 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:42.290325 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:42.290449 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 02:56:42.290493 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:56:42.291481 32496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:42.284496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:40109 (local address 127.12.222.195:49079)
0925 02:56:42.285075 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:42.285109 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:42.286363 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:42.286853 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:42.286861 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:42.286964 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:42.287366 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:42.287377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.288169 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.288174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:42.289003 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:42.289013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.289415 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.289418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:42.289545 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:42.289992 (+ 447us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:56:42.289994 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:42.290027 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:56:42.290079 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:42.290336 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:42.290342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:42.290345 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:56:42.290428 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 02:56:42.290637 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":110,"threads_started":1}
I20260925 02:56:42.292500 32494 heartbeater.cc:347] Connected to a master server at 127.12.222.254:40109
I20260925 02:56:42.292721 32494 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:42.293188 32494 heartbeater.cc:511] Master 127.12.222.254:40109 requested a full tablet report, sending...
I20260925 02:56:42.294138 32038 ts_manager.cc:194] Registered new tserver with Master: 025155ad33054ce7af52c5376cbecf16 (127.12.222.195:46125)
I20260925 02:56:42.294687 32038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.222.195:49079
I20260925 02:56:42.299698 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:56:42.304853 32078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:42.301532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42962 (local address 127.12.222.254:40109)
0925 02:56:42.301657 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:42.301661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:42.301838 (+ 177us) server_negotiation.cc:407] Connection header received
0925 02:56:42.301944 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:42.301946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:42.301979 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:42.302050 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:42.302277 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.302699 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.303397 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.303609 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.303678 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:42.304399 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:42.304401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:42.304403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:42.304413 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:42.304428 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:42.304435 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:42.304528 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:42.304655 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 02:56:42.304719 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:56:42.307222 32038 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:56:42.310073 32038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:56:42.322602 32506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:42.317341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.12.222.195:46125)
0925 02:56:42.317704 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:42.317714 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:42.317737 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:56:42.317982 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:42.317988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:42.318179 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:42.318332 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:42.318739 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.319660 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.320837 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.321771 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.321885 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:42.322058 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:42.322355 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 02:56:42.322439 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":101,"threads_started":1}
I20260925 02:56:42.322590 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:42.317220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:46125 (local address 127.0.0.1:52230)
0925 02:56:42.317593 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:42.317641 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:42.317789 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:42.318459 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:42.318464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:42.318482 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:42.318614 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:42.318620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.319758 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.319764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:42.320707 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:42.320716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.322115 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.322117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:42.322188 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:42.322213 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:56:42.322315 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 02:56:42.322435 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":75,"threads_started":1}
I20260925 02:56:42.323549 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:42.317720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:45609 (local address 127.0.0.1:55150)
0925 02:56:42.318009 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:42.318020 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:42.318091 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:42.319250 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:42.319253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:42.319273 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:42.319360 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:42.319366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.320785 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.320791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:42.321900 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:42.321911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.323152 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.323155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:42.323238 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:42.323260 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:56:42.323336 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 02:56:42.323401 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":98,"threads_started":1}
I20260925 02:56:42.323771 32509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:42.317969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55150 (local address 127.12.222.194:45609)
0925 02:56:42.318747 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:42.318758 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:42.318776 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:56:42.318834 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:42.318837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:42.319115 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:42.319520 (+ 405us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:42.319644 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.320672 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.322033 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.323040 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.323155 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:42.323259 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:42.323377 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 02:56:42.323485 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":105,"threads_started":1}
I20260925 02:56:42.324043 32510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:42.318298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49174 (local address 127.12.222.193:39699)
0925 02:56:42.318707 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:42.318718 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:42.318740 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:56:42.318815 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:42.318819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:42.319005 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:42.319149 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:42.319362 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.320481 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.321847 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.322604 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.322877 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:42.323035 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:42.323794 (+ 759us) server_negotiation.cc:299] Negotiation successful
0925 02:56:42.323904 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":103,"threads_started":1}
I20260925 02:56:42.324327 32508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:42.317904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:39699 (local address 127.0.0.1:49174)
0925 02:56:42.318272 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:42.318283 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:42.318349 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:42.319142 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:42.319145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:42.319163 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:42.319245 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:42.319250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.320597 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.320600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:42.321730 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:42.321741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.323338 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.323344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:42.323432 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:42.323454 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:56:42.323548 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:56:42.324186 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":48,"thread_start_us":90,"threads_started":1}
I20260925 02:56:42.325644 32427 tablet_service.cc:1467] Processing CreateTablet for tablet 280f09e5ce84428ca4defab0f164d4f1 (DEFAULT_TABLE table=test-table [id=dd0991ff95f243289448d7615bc68f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:42.326668 32427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 280f09e5ce84428ca4defab0f164d4f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:42.326830 32162 tablet_service.cc:1467] Processing CreateTablet for tablet 280f09e5ce84428ca4defab0f164d4f1 (DEFAULT_TABLE table=test-table [id=dd0991ff95f243289448d7615bc68f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:42.327723 32162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 280f09e5ce84428ca4defab0f164d4f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:42.330248 32294 tablet_service.cc:1467] Processing CreateTablet for tablet 280f09e5ce84428ca4defab0f164d4f1 (DEFAULT_TABLE table=test-table [id=dd0991ff95f243289448d7615bc68f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:42.330991 32294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 280f09e5ce84428ca4defab0f164d4f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:42.333710 32511 tablet_bootstrap.cc:492] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16: Bootstrap starting.
I20260925 02:56:42.334935 32512 tablet_bootstrap.cc:492] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1: Bootstrap starting.
I20260925 02:56:42.335983 32511 tablet_bootstrap.cc:654] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:42.336643 32512 tablet_bootstrap.cc:654] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:42.336879 32511 log.cc:824] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:42.337461 32512 log.cc:824] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:42.337730 32514 tablet_bootstrap.cc:492] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07: Bootstrap starting.
I20260925 02:56:42.338738 32511 tablet_bootstrap.cc:492] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16: No bootstrap required, opened a new log
I20260925 02:56:42.338936 32511 ts_tablet_manager.cc:1397] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260925 02:56:42.339357 32512 tablet_bootstrap.cc:492] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1: No bootstrap required, opened a new log
I20260925 02:56:42.339524 32512 ts_tablet_manager.cc:1397] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:56:42.339754 32514 tablet_bootstrap.cc:654] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:42.340545 32514 log.cc:824] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:42.342303 32511 raft_consensus.cc:348] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025155ad33054ce7af52c5376cbecf16" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46125 } } peers { permanent_uuid: "6983bcbdaa0641e8923c39f9edb6cb07" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 45609 } } peers { permanent_uuid: "2019fdbda6db44c6989f108f9d81c8b1" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39699 } }
I20260925 02:56:42.342439 32514 tablet_bootstrap.cc:492] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07: No bootstrap required, opened a new log
I20260925 02:56:42.342518 32511 raft_consensus.cc:374] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:42.342578 32511 raft_consensus.cc:729] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025155ad33054ce7af52c5376cbecf16, State: Initialized, Role: FOLLOWER
I20260925 02:56:42.342618 32514 ts_tablet_manager.cc:1397] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:56:42.342998 32511 consensus_queue.cc:260] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025155ad33054ce7af52c5376cbecf16" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46125 } } peers { permanent_uuid: "6983bcbdaa0641e8923c39f9edb6cb07" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 45609 } } peers { permanent_uuid: "2019fdbda6db44c6989f108f9d81c8b1" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39699 } }
I20260925 02:56:42.343595 32494 heartbeater.cc:503] Master 127.12.222.254:40109 was elected leader, sending a full tablet report...
I20260925 02:56:42.343511 32512 raft_consensus.cc:348] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025155ad33054ce7af52c5376cbecf16" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46125 } } peers { permanent_uuid: "6983bcbdaa0641e8923c39f9edb6cb07" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 45609 } } peers { permanent_uuid: "2019fdbda6db44c6989f108f9d81c8b1" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39699 } }
I20260925 02:56:42.343750 32512 raft_consensus.cc:374] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:42.343809 32512 raft_consensus.cc:729] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2019fdbda6db44c6989f108f9d81c8b1, State: Initialized, Role: FOLLOWER
I20260925 02:56:42.343951 32511 ts_tablet_manager.cc:1428] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260925 02:56:42.344213 32512 consensus_queue.cc:260] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025155ad33054ce7af52c5376cbecf16" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46125 } } peers { permanent_uuid: "6983bcbdaa0641e8923c39f9edb6cb07" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 45609 } } peers { permanent_uuid: "2019fdbda6db44c6989f108f9d81c8b1" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39699 } }
I20260925 02:56:42.345150 32512 ts_tablet_manager.cc:1428] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260925 02:56:42.346200 32228 heartbeater.cc:503] Master 127.12.222.254:40109 was elected leader, sending a full tablet report...
I20260925 02:56:42.346638 32514 raft_consensus.cc:348] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025155ad33054ce7af52c5376cbecf16" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46125 } } peers { permanent_uuid: "6983bcbdaa0641e8923c39f9edb6cb07" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 45609 } } peers { permanent_uuid: "2019fdbda6db44c6989f108f9d81c8b1" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39699 } }
I20260925 02:56:42.346885 32514 raft_consensus.cc:374] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:42.346941 32514 raft_consensus.cc:729] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6983bcbdaa0641e8923c39f9edb6cb07, State: Initialized, Role: FOLLOWER
I20260925 02:56:42.347337 32514 consensus_queue.cc:260] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025155ad33054ce7af52c5376cbecf16" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46125 } } peers { permanent_uuid: "6983bcbdaa0641e8923c39f9edb6cb07" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 45609 } } peers { permanent_uuid: "2019fdbda6db44c6989f108f9d81c8b1" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39699 } }
I20260925 02:56:42.347967 32361 heartbeater.cc:503] Master 127.12.222.254:40109 was elected leader, sending a full tablet report...
I20260925 02:56:42.348217 32514 ts_tablet_manager.cc:1428] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260925 02:56:42.371263 32362 tablet.cc:2369] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:42.424551 32518 raft_consensus.cc:482] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:56:42.424790 32518 raft_consensus.cc:504] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025155ad33054ce7af52c5376cbecf16" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46125 } } peers { permanent_uuid: "6983bcbdaa0641e8923c39f9edb6cb07" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 45609 } } peers { permanent_uuid: "2019fdbda6db44c6989f108f9d81c8b1" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39699 } }
I20260925 02:56:42.425904 32518 leader_election.cc:288] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 025155ad33054ce7af52c5376cbecf16 (127.12.222.195:46125), 6983bcbdaa0641e8923c39f9edb6cb07 (127.12.222.194:45609)
I20260925 02:56:42.429414 32506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:42.426201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:56973 (local address 127.12.222.195:46125)
0925 02:56:42.426334 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:42.426338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:42.426356 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:56:42.426490 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:42.426495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:42.426550 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:42.426626 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:42.426899 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.427436 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.428493 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.428873 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.429007 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:42.429051 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:42.429224 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 02:56:42.429292 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:56:42.429409 32230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:42.426090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:46125 (local address 127.12.222.193:56973)
0925 02:56:42.426264 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:42.426278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:42.426372 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:42.426648 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:42.426651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:42.426668 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:42.426789 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:42.426796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.427536 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.427541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:42.428380 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:42.428389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.429003 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.429006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:42.429115 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:42.429131 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:56:42.429187 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 02:56:42.429265 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":2}
I20260925 02:56:42.429426 32520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:42.426112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:45609 (local address 127.12.222.193:59385)
0925 02:56:42.426460 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:42.426473 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:42.426558 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:42.426802 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:42.426805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:42.426818 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:42.426961 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:42.426967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.427619 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.427622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:42.428408 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:42.428417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.429007 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.429011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:42.429134 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:42.429150 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:56:42.429236 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 02:56:42.429294 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":49,"thread_start_us":93,"threads_started":1}
I20260925 02:56:42.429440 32509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:42.426212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:59385 (local address 127.12.222.194:45609)
0925 02:56:42.426323 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:42.426326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:42.426547 (+ 221us) server_negotiation.cc:407] Connection header received
0925 02:56:42.426657 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:42.426659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:42.426705 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:42.426780 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:42.427063 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.427518 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.428537 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.428881 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.429004 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:42.429051 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:42.429267 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 02:56:42.429332 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:56:42.430087 32447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280f09e5ce84428ca4defab0f164d4f1" candidate_uuid: "2019fdbda6db44c6989f108f9d81c8b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025155ad33054ce7af52c5376cbecf16" is_pre_election: true
I20260925 02:56:42.430269 32314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280f09e5ce84428ca4defab0f164d4f1" candidate_uuid: "2019fdbda6db44c6989f108f9d81c8b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6983bcbdaa0641e8923c39f9edb6cb07" is_pre_election: true
I20260925 02:56:42.430387 32447 raft_consensus.cc:2454] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2019fdbda6db44c6989f108f9d81c8b1 in term 0.
I20260925 02:56:42.430526 32314 raft_consensus.cc:2454] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2019fdbda6db44c6989f108f9d81c8b1 in term 0.
I20260925 02:56:42.430912 32118 leader_election.cc:302] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 025155ad33054ce7af52c5376cbecf16, 2019fdbda6db44c6989f108f9d81c8b1, 6983bcbdaa0641e8923c39f9edb6cb07; no voters:
I20260925 02:56:42.431181 32518 raft_consensus.cc:2781] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:56:42.431263 32518 raft_consensus.cc:482] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:56:42.431303 32518 raft_consensus.cc:3037] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:42.432322 32518 raft_consensus.cc:504] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025155ad33054ce7af52c5376cbecf16" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46125 } } peers { permanent_uuid: "6983bcbdaa0641e8923c39f9edb6cb07" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 45609 } } peers { permanent_uuid: "2019fdbda6db44c6989f108f9d81c8b1" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39699 } }
I20260925 02:56:42.432677 32518 leader_election.cc:288] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [CANDIDATE]: Term 1 election: Requested vote from peers 025155ad33054ce7af52c5376cbecf16 (127.12.222.195:46125), 6983bcbdaa0641e8923c39f9edb6cb07 (127.12.222.194:45609)
I20260925 02:56:42.432955 32447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280f09e5ce84428ca4defab0f164d4f1" candidate_uuid: "2019fdbda6db44c6989f108f9d81c8b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025155ad33054ce7af52c5376cbecf16"
I20260925 02:56:42.433032 32314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280f09e5ce84428ca4defab0f164d4f1" candidate_uuid: "2019fdbda6db44c6989f108f9d81c8b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6983bcbdaa0641e8923c39f9edb6cb07"
I20260925 02:56:42.433089 32447 raft_consensus.cc:3037] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:42.433153 32314 raft_consensus.cc:3037] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:42.434115 32447 raft_consensus.cc:2454] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2019fdbda6db44c6989f108f9d81c8b1 in term 1.
I20260925 02:56:42.434242 32314 raft_consensus.cc:2454] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2019fdbda6db44c6989f108f9d81c8b1 in term 1.
I20260925 02:56:42.434413 32118 leader_election.cc:302] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025155ad33054ce7af52c5376cbecf16, 2019fdbda6db44c6989f108f9d81c8b1; no voters:
I20260925 02:56:42.434681 32518 raft_consensus.cc:2781] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:42.434897 32518 raft_consensus.cc:686] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [term 1 LEADER]: Becoming Leader. State: Replica: 2019fdbda6db44c6989f108f9d81c8b1, State: Running, Role: LEADER
I20260925 02:56:42.435187 32518 consensus_queue.cc:237] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025155ad33054ce7af52c5376cbecf16" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46125 } } peers { permanent_uuid: "6983bcbdaa0641e8923c39f9edb6cb07" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 45609 } } peers { permanent_uuid: "2019fdbda6db44c6989f108f9d81c8b1" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39699 } }
I20260925 02:56:42.438220 32037 catalog_manager.cc:5495] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2019fdbda6db44c6989f108f9d81c8b1 (127.12.222.193). New cstate: current_term: 1 leader_uuid: "2019fdbda6db44c6989f108f9d81c8b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025155ad33054ce7af52c5376cbecf16" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6983bcbdaa0641e8923c39f9edb6cb07" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 45609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2019fdbda6db44c6989f108f9d81c8b1" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 39699 } health_report { overall_health: HEALTHY } } }
W20260925 02:56:42.453693 32229 tablet.cc:2369] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:42.474366 32510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:42.470859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.12.222.193:39699)
0925 02:56:42.470997 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:42.471002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:42.471016 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:56:42.471170 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:42.471172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:42.471223 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:42.471324 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:42.471600 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.472077 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.472963 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:42.473158 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:42.473258 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:42.473562 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:42.473931 (+ 369us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:42.474095 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:42.474187 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 02:56:42.474221 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:56:42.479959 32447 raft_consensus.cc:1261] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16 [term 1 FOLLOWER]: Refusing update from remote peer 2019fdbda6db44c6989f108f9d81c8b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:42.479959 32314 raft_consensus.cc:1261] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07 [term 1 FOLLOWER]: Refusing update from remote peer 2019fdbda6db44c6989f108f9d81c8b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:42.480710 32521 consensus_queue.cc:1035] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6983bcbdaa0641e8923c39f9edb6cb07" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 45609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:42.480962 32518 consensus_queue.cc:1035] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025155ad33054ce7af52c5376cbecf16" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 46125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:42.487432 32524 mvcc.cc:205] Tried to move back new op lower bound from 7333089290149974016 to 7333089289976856576. Current Snapshot: MvccSnapshot[applied={T|T < 7333089290149974016}]
I20260925 02:56:42.495942 32037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:42.496061 32037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:42.498148 32037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f6d7badfd0f4c858518a3d6952f8c51: Sending DeleteTablet for 3 replicas of tablet 280f09e5ce84428ca4defab0f164d4f1
I20260925 02:56:42.498867 32294 tablet_service.cc:1514] Processing DeleteTablet for tablet 280f09e5ce84428ca4defab0f164d4f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:42 UTC) from {username='slave'} at 127.0.0.1:55150
I20260925 02:56:42.498895 32427 tablet_service.cc:1514] Processing DeleteTablet for tablet 280f09e5ce84428ca4defab0f164d4f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:42 UTC) from {username='slave'} at 127.0.0.1:52230
I20260925 02:56:42.499017 32162 tablet_service.cc:1514] Processing DeleteTablet for tablet 280f09e5ce84428ca4defab0f164d4f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:42 UTC) from {username='slave'} at 127.0.0.1:49174
I20260925 02:56:42.499301 32534 tablet_replica.cc:330] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07: stopping tablet replica
I20260925 02:56:42.499400 32536 tablet_replica.cc:330] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1: stopping tablet replica
I20260925 02:56:42.499589 32534 raft_consensus.cc:2229] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:42.499660 32536 raft_consensus.cc:2229] T 280f09e5ce84428ca4defab0f164d4f1 P 2019fdbda6db44c6989f108f9d81c8b1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:56:42.499797 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 32099
I20260925 02:56:42.499873 32534 raft_consensus.cc:2258] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:42.500542 32535 tablet_replica.cc:330] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16: stopping tablet replica
I20260925 02:56:42.500759 32535 raft_consensus.cc:2229] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:42.501026 32535 raft_consensus.cc:2258] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:42.502517 32534 ts_tablet_manager.cc:1905] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:42.504802 32534 ts_tablet_manager.cc:1918] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:56:42.504906 32534 log.cc:1196] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/280f09e5ce84428ca4defab0f164d4f1
I20260925 02:56:42.505194 32534 ts_tablet_manager.cc:1939] T 280f09e5ce84428ca4defab0f164d4f1 P 6983bcbdaa0641e8923c39f9edb6cb07: Deleting consensus metadata
I20260925 02:56:42.505496 32535 ts_tablet_manager.cc:1905] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:42.506311 32022 catalog_manager.cc:4841] TS 6983bcbdaa0641e8923c39f9edb6cb07 (127.12.222.194:45609): tablet 280f09e5ce84428ca4defab0f164d4f1 (table test-table [id=dd0991ff95f243289448d7615bc68f71]) successfully deleted
W20260925 02:56:42.506829 32025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.222.193:39699 (error 108)
I20260925 02:56:42.507001 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 32231
I20260925 02:56:42.508157 32535 ts_tablet_manager.cc:1918] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:56:42.508265 32535 log.cc:1196] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/280f09e5ce84428ca4defab0f164d4f1
I20260925 02:56:42.508420 32508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:42.508127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:39699 (local address 127.0.0.1:49180)
0925 02:56:42.508272 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:42.508341 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.193:39699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 02:56:42.508551 32535 ts_tablet_manager.cc:1939] T 280f09e5ce84428ca4defab0f164d4f1 P 025155ad33054ce7af52c5376cbecf16: Deleting consensus metadata
W20260925 02:56:42.508591 32025 catalog_manager.cc:4584] TS 2019fdbda6db44c6989f108f9d81c8b1 (127.12.222.193:39699): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 280f09e5ce84428ca4defab0f164d4f1: Network error: Client connection negotiation failed: client connection to 127.12.222.193:39699: connect: Connection refused (error 111)
I20260925 02:56:42.509598 32025 catalog_manager.cc:4841] TS 025155ad33054ce7af52c5376cbecf16 (127.12.222.195:46125): tablet 280f09e5ce84428ca4defab0f164d4f1 (table test-table [id=dd0991ff95f243289448d7615bc68f71]) successfully deleted
I20260925 02:56:42.513512 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 32364
I20260925 02:56:42.518822 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 32007
2026-09-25T02:56:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1020 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:42 dist-test-slave-252r krb5kdc[32540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:42 dist-test-slave-252r krb5kdc[32540](info): set up 2 sockets
Sep 25 02:56:42 dist-test-slave-252r krb5kdc[32540](info): commencing operation
krb5kdc: starting...
W20260925 02:56:43.475449 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:56:43 dist-test-slave-252r krb5kdc[32540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:56:43.580761 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:33001
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:33001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:43.685348 32553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:43.685559 32553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:43.685595 32553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:43.689219 32553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:43.689265 32553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:43.689284 32553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:43.689299 32553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:43.689319 32553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:43.694645 32553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:33001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:33001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:43.695516 32553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:43.696446 32553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:43.704064 32558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:43.704048 32559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:43.704113 32553 server_base.cc:1034] running on GCE node
W20260925 02:56:43.704061 32561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:43.704866 32553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:43.705967 32553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:43.707149 32553 hybrid_clock.cc:648] HybridClock initialized: now 1790305003707126 us; error 44 us; skew 500 ppm
Sep 25 02:56:43 dist-test-slave-252r krb5kdc[32540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305003, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:43.709631 32553 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:56:43.710448 32553 webserver.cc:460] Webserver started at http://127.12.222.254:44629/ using document root <none> and password file <none>
I20260925 02:56:43.710945 32553 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:43.711031 32553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:43.711251 32553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:43.713130 32553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "f525a1e0c8ef48b9887b9f073c59609d"
format_stamp: "Formatted at 2026-09-25 02:56:43 on dist-test-slave-252r"
server_key: "b96068dfb3badd95850fe73ddf1097f8"
server_key_iv: "c5f53c16d6a294d6d7b833c7b0be5a43"
server_key_version: "encryptionkey@0"
I20260925 02:56:43.713680 32553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "f525a1e0c8ef48b9887b9f073c59609d"
format_stamp: "Formatted at 2026-09-25 02:56:43 on dist-test-slave-252r"
server_key: "b96068dfb3badd95850fe73ddf1097f8"
server_key_iv: "c5f53c16d6a294d6d7b833c7b0be5a43"
server_key_version: "encryptionkey@0"
I20260925 02:56:43.716964 32553 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:56:43.719128 32568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:43.720270 32553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:43.720402 32553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "f525a1e0c8ef48b9887b9f073c59609d"
format_stamp: "Formatted at 2026-09-25 02:56:43 on dist-test-slave-252r"
server_key: "b96068dfb3badd95850fe73ddf1097f8"
server_key_iv: "c5f53c16d6a294d6d7b833c7b0be5a43"
server_key_version: "encryptionkey@0"
I20260925 02:56:43.720501 32553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:43.730618 32553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:43.732980 32553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:43.733150 32553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:43.739511 32553 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:33001
I20260925 02:56:43.739540 32622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:33001 every 8 connection(s)
I20260925 02:56:43.740330 32553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:43.742574 32623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:43.745358 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 32553
I20260925 02:56:43.745509 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:56:43.745806 13179 external_mini_cluster.cc:1428] Setting key 934a42f59990f7bfaf25cd17f53abdd2
I20260925 02:56:43.748216 32623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d: Bootstrap starting.
I20260925 02:56:43.750203 32623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:43.750921 32623 log.cc:824] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:43.752697 32623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d: No bootstrap required, opened a new log
Sep 25 02:56:43 dist-test-slave-252r krb5kdc[32540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:56:43.756037 32623 raft_consensus.cc:348] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f525a1e0c8ef48b9887b9f073c59609d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33001 } }
I20260925 02:56:43.756299 32623 raft_consensus.cc:374] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:43.756357 32623 raft_consensus.cc:729] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f525a1e0c8ef48b9887b9f073c59609d, State: Initialized, Role: FOLLOWER
I20260925 02:56:43.756831 32623 consensus_queue.cc:260] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f525a1e0c8ef48b9887b9f073c59609d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33001 } }
I20260925 02:56:43.757002 32623 raft_consensus.cc:388] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:43.757061 32623 raft_consensus.cc:482] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:43.757151 32623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:43.757856 32623 raft_consensus.cc:504] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f525a1e0c8ef48b9887b9f073c59609d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33001 } }
I20260925 02:56:43.758200 32623 leader_election.cc:302] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f525a1e0c8ef48b9887b9f073c59609d; no voters:
I20260925 02:56:43.758474 32623 leader_election.cc:288] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:43.758558 32629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:43.758773 32629 raft_consensus.cc:686] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d [term 1 LEADER]: Becoming Leader. State: Replica: f525a1e0c8ef48b9887b9f073c59609d, State: Running, Role: LEADER
I20260925 02:56:43.758917 32627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:43.747111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56528 (local address 127.12.222.254:33001)
0925 02:56:43.747509 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:43.747522 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:43.747553 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:56:43.748142 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:43.748157 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:43.748545 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:43.748858 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:43.749662 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.750638 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.751539 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.751988 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.752143 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:43.754388 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:43.754393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:43.754404 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:43.754428 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:43.755976 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:43.756477 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:43.756483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:43.756486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:43.756618 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:43.756873 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:43.756876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:43.756878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:43.757340 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:43.757535 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:43.757802 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 02:56:43.758071 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":126,"threads_started":1}
I20260925 02:56:43.759049 32629 consensus_queue.cc:237] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f525a1e0c8ef48b9887b9f073c59609d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33001 } }
I20260925 02:56:43.759570 32623 sys_catalog.cc:564] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:43.760077 32631 sys_catalog.cc:455] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f525a1e0c8ef48b9887b9f073c59609d. Latest consensus state: current_term: 1 leader_uuid: "f525a1e0c8ef48b9887b9f073c59609d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f525a1e0c8ef48b9887b9f073c59609d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33001 } } }
I20260925 02:56:43.760205 32631 sys_catalog.cc:458] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:43.760944 32635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:43.761848 32630 sys_catalog.cc:455] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f525a1e0c8ef48b9887b9f073c59609d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f525a1e0c8ef48b9887b9f073c59609d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33001 } } }
I20260925 02:56:43.761953 32630 sys_catalog.cc:458] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:43.763517 32635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:43.768481 32635 catalog_manager.cc:1347] Generated new cluster ID: 29ab71bbf9694f329bd063104ef4a312
I20260925 02:56:43.768536 32635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:43.790361 32635 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:43.791291 32635 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:43.796196 32635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d: Generated new TSK 0
I20260925 02:56:43.796792 32635 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.222.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
Entry for principal kudu/127.12.222.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab.
I20260925 02:56:43.825404 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:33001
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:43.924961 32650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:43.925196 32650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:43.925264 32650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:43.928884 32650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:43.928960 32650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:43.929042 32650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:56:43.933729 32650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.193.keytab
--principal=kudu/127.12.222.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:33001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:43.934589 32650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:43.935477 32650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:56:43.943769 32650 server_base.cc:1034] running on GCE node
W20260925 02:56:43.943773 32658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:43.944001 32655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:43.944026 32656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:43.944425 32650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:43.944998 32650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:43.946151 32650 hybrid_clock.cc:648] HybridClock initialized: now 1790305003946132 us; error 28 us; skew 500 ppm
Sep 25 02:56:43 dist-test-slave-252r krb5kdc[32540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305003, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:43.948674 32650 init.cc:377] Logged in from keytab as kudu/127.12.222.193@KRBTEST.COM (short username kudu)
I20260925 02:56:43.949461 32650 webserver.cc:460] Webserver started at http://127.12.222.193:40181/ using document root <none> and password file <none>
I20260925 02:56:43.949967 32650 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:43.950053 32650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:43.950292 32650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:43.952162 32650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "a6d40d6338af44389609dc4d7e5edf71"
format_stamp: "Formatted at 2026-09-25 02:56:43 on dist-test-slave-252r"
server_key: "cfdf07249105f875cc79b7b219b16af9"
server_key_iv: "941c3e6b16c506aa7c15453895ecc3f1"
server_key_version: "encryptionkey@0"
I20260925 02:56:43.952716 32650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "a6d40d6338af44389609dc4d7e5edf71"
format_stamp: "Formatted at 2026-09-25 02:56:43 on dist-test-slave-252r"
server_key: "cfdf07249105f875cc79b7b219b16af9"
server_key_iv: "941c3e6b16c506aa7c15453895ecc3f1"
server_key_version: "encryptionkey@0"
I20260925 02:56:43.956054 32650 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:56:43.958087 32665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:43.959115 32650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:43.959235 32650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "a6d40d6338af44389609dc4d7e5edf71"
format_stamp: "Formatted at 2026-09-25 02:56:43 on dist-test-slave-252r"
server_key: "cfdf07249105f875cc79b7b219b16af9"
server_key_iv: "941c3e6b16c506aa7c15453895ecc3f1"
server_key_version: "encryptionkey@0"
I20260925 02:56:43.959344 32650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:43.974174 32650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:43.976588 32650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:43.976760 32650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:43.977258 32650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:43.977954 32650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:43.978027 32650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:43.978084 32650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:43.978128 32650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:43.986157 32650 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:41347
I20260925 02:56:43.986182 313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:41347 every 8 connection(s)
I20260925 02:56:43.986902 32650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:56:43.990170 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 32650
I20260925 02:56:43.990298 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:56:43.990602 13179 external_mini_cluster.cc:1428] Setting key e5f52d0ebb2fd25fe6539d98339b40d3
Sep 25 02:56:43 dist-test-slave-252r krb5kdc[32540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305003, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.193@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.222.194@KRBTEST.COM; defaulting to no policy
I20260925 02:56:44.000597 32627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:43.988426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:47739 (local address 127.12.222.254:33001)
0925 02:56:43.988544 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:43.988548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:43.989281 (+ 733us) server_negotiation.cc:407] Connection header received
0925 02:56:43.990800 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:43.990802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:43.990845 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:43.990911 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:43.992124 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.992642 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.994017 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.994407 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.994524 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:43.997131 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:43.997133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:43.997135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:43.997156 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:43.998489 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:43.999072 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:43.999075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:43.999076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:43.999147 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:43.999537 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:43.999539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:43.999540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:43.999694 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:43.999781 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.000326 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.000444 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:56:44.001289 316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:43.988773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:33001 (local address 127.12.222.193:47739)
0925 02:56:43.989156 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:43.989191 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:43.990524 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:43.991098 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:43.991105 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:43.991471 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:43.991981 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:43.992000 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.992759 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.992764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:43.993901 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:43.993911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:43.994546 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:43.994549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:43.994679 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:43.995238 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:43.995252 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:43.996974 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:43.998609 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:43.998617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:43.998632 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:43.998973 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:43.999245 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:43.999250 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:43.999254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:43.999440 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:43.999797 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:43.999805 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:44.000086 (+ 281us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.000292 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.000564 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":102,"threads_started":1}
I20260925 02:56:44.002400 314 heartbeater.cc:347] Connected to a master server at 127.12.222.254:33001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.194@KRBTEST.COM" created.
I20260925 02:56:44.002645 314 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:44.003127 314 heartbeater.cc:511] Master 127.12.222.254:33001 requested a full tablet report, sending...
I20260925 02:56:44.004541 32586 ts_manager.cc:194] Registered new tserver with Master: a6d40d6338af44389609dc4d7e5edf71 (127.12.222.193:41347)
I20260925 02:56:44.005570 32586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.193@KRBTEST.COM'} at 127.12.222.193:47739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
Entry for principal kudu/127.12.222.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab.
I20260925 02:56:44.017258 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:33001
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:44.113204 319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:44.113436 319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:44.113498 319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:44.117063 319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:44.117125 319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:44.117197 319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:56:44.121703 319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.194.keytab
--principal=kudu/127.12.222.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:33001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:44.122512 319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:44.123359 319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:44.131338 324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:44.131397 319 server_base.cc:1034] running on GCE node
W20260925 02:56:44.131531 327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:44.131417 325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:44.131961 319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:44.132534 319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:44.133725 319 hybrid_clock.cc:648] HybridClock initialized: now 1790305004133710 us; error 25 us; skew 500 ppm
Sep 25 02:56:44 dist-test-slave-252r krb5kdc[32540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305004, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:44.136104 319 init.cc:377] Logged in from keytab as kudu/127.12.222.194@KRBTEST.COM (short username kudu)
I20260925 02:56:44.136802 319 webserver.cc:460] Webserver started at http://127.12.222.194:35921/ using document root <none> and password file <none>
I20260925 02:56:44.137220 319 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:44.137271 319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:44.137403 319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:44.139021 319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "fd9ad1b544154db699127c7a1b9ca1bf"
format_stamp: "Formatted at 2026-09-25 02:56:44 on dist-test-slave-252r"
server_key: "deada36525d10e93f1d46d63e9e4facd"
server_key_iv: "42b401c8100641ea218f9214cbc4a732"
server_key_version: "encryptionkey@0"
I20260925 02:56:44.139454 319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "fd9ad1b544154db699127c7a1b9ca1bf"
format_stamp: "Formatted at 2026-09-25 02:56:44 on dist-test-slave-252r"
server_key: "deada36525d10e93f1d46d63e9e4facd"
server_key_iv: "42b401c8100641ea218f9214cbc4a732"
server_key_version: "encryptionkey@0"
I20260925 02:56:44.142663 319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 02:56:44.144711 334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:44.145725 319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:44.145808 319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "fd9ad1b544154db699127c7a1b9ca1bf"
format_stamp: "Formatted at 2026-09-25 02:56:44 on dist-test-slave-252r"
server_key: "deada36525d10e93f1d46d63e9e4facd"
server_key_iv: "42b401c8100641ea218f9214cbc4a732"
server_key_version: "encryptionkey@0"
I20260925 02:56:44.145879 319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:44.160938 319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:44.163484 319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:44.163614 319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:44.164219 319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:44.165002 319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:44.165050 319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:44.165083 319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:44.165100 319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:44.172461 319 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:34739
I20260925 02:56:44.172502 448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:34739 every 8 connection(s)
I20260925 02:56:44.173143 319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:56:44 dist-test-slave-252r krb5kdc[32540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305004, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.194@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:56:44.182251 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 319
I20260925 02:56:44.182372 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:56:44.182642 13179 external_mini_cluster.cc:1428] Setting key f487894f0ffb24b9dbfe4749c3ced0e7
I20260925 02:56:44.184787 32627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.174582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:51853 (local address 127.12.222.254:33001)
0925 02:56:44.174729 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.174734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.175432 (+ 698us) server_negotiation.cc:407] Connection header received
0925 02:56:44.176546 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.176549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.176591 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.176658 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:44.177560 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.178079 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.179053 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.179242 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.179308 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.181357 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:44.181360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:44.181361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:44.181382 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:44.182742 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:44.183288 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:44.183290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:44.183291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:44.183357 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:44.183693 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:44.183697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:44.183698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:44.183920 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:44.184016 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.184579 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.184668 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:56:44.185748 451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.174899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:33001 (local address 127.12.222.194:51853)
0925 02:56:44.175296 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.175332 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.176347 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.176793 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.176799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.177071 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:44.177451 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.177462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.178171 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.178173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.178961 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.178969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.179347 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.179349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.179473 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.179956 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:44.179969 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:44.181223 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:44.182850 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:44.182855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:44.182866 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:44.183188 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:44.183471 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:44.183473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:44.183474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:44.183583 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:44.184031 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:44.184036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:44.184343 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.184547 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.184867 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":109,"threads_started":1}
I20260925 02:56:44.186684 449 heartbeater.cc:347] Connected to a master server at 127.12.222.254:33001
I20260925 02:56:44.186908 449 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:44.187376 449 heartbeater.cc:511] Master 127.12.222.254:33001 requested a full tablet report, sending...
I20260925 02:56:44.188424 32586 ts_manager.cc:194] Registered new tserver with Master: fd9ad1b544154db699127c7a1b9ca1bf (127.12.222.194:34739)
I20260925 02:56:44.188932 32586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.194@KRBTEST.COM'} at 127.12.222.194:51853
WARNING: no policy specified for kudu/127.12.222.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
Entry for principal kudu/127.12.222.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab.
I20260925 02:56:44.206835 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:33001
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:44.306574 454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:44.306802 454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:44.306855 454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:44.310676 454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:44.310720 454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:44.310770 454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:56:44.315579 454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.195.keytab
--principal=kudu/127.12.222.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:33001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:44.316450 454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:44.317276 454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:44.325834 462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:44.325909 460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:44.325824 459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:44.325997 454 server_base.cc:1034] running on GCE node
I20260925 02:56:44.326399 454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:44.326938 454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:44.328127 454 hybrid_clock.cc:648] HybridClock initialized: now 1790305004328098 us; error 34 us; skew 500 ppm
Sep 25 02:56:44 dist-test-slave-252r krb5kdc[32540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305004, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:44.330585 454 init.cc:377] Logged in from keytab as kudu/127.12.222.195@KRBTEST.COM (short username kudu)
I20260925 02:56:44.331296 454 webserver.cc:460] Webserver started at http://127.12.222.195:42225/ using document root <none> and password file <none>
I20260925 02:56:44.331724 454 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:44.331774 454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:44.331919 454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:44.333709 454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "139af6d18f1c419691e05bf40beb3526"
format_stamp: "Formatted at 2026-09-25 02:56:44 on dist-test-slave-252r"
server_key: "0a7007b50625f234fbf4853c697b53e0"
server_key_iv: "7dbe2b9547e02e03df4caca641f19513"
server_key_version: "encryptionkey@0"
I20260925 02:56:44.334187 454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "139af6d18f1c419691e05bf40beb3526"
format_stamp: "Formatted at 2026-09-25 02:56:44 on dist-test-slave-252r"
server_key: "0a7007b50625f234fbf4853c697b53e0"
server_key_iv: "7dbe2b9547e02e03df4caca641f19513"
server_key_version: "encryptionkey@0"
I20260925 02:56:44.337319 454 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 02:56:44.339320 469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:44.340502 454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:56:44.340632 454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "139af6d18f1c419691e05bf40beb3526"
format_stamp: "Formatted at 2026-09-25 02:56:44 on dist-test-slave-252r"
server_key: "0a7007b50625f234fbf4853c697b53e0"
server_key_iv: "7dbe2b9547e02e03df4caca641f19513"
server_key_version: "encryptionkey@0"
I20260925 02:56:44.340730 454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:44.354197 454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:44.356560 454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:44.356724 454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:44.357206 454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:44.357924 454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:44.357995 454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:44.358052 454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:44.358094 454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:44.366659 454 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:37981
I20260925 02:56:44.366684 583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:37981 every 8 connection(s)
I20260925 02:56:44.367372 454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:56:44.371734 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 454
I20260925 02:56:44.371878 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:56:44.372141 13179 external_mini_cluster.cc:1428] Setting key 205a2d9f2c0fd81ed1deaf16435179ca
Sep 25 02:56:44 dist-test-slave-252r krb5kdc[32540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305004, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.195@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:56:44.379673 32627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.369011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:35877 (local address 127.12.222.254:33001)
0925 02:56:44.369164 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.369168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.369906 (+ 738us) server_negotiation.cc:407] Connection header received
0925 02:56:44.371096 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.371098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.371159 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.371242 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:44.372328 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.373001 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.374068 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.374293 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.374414 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.376650 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:44.376653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:44.376656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:44.376685 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:44.377794 (+ 1109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:44.378313 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:44.378316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:44.378318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:44.378402 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:44.378729 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:44.378733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:44.378734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:44.378901 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:44.379000 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.379416 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.379527 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:56:44.380398 586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.369391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:33001 (local address 127.12.222.195:35877)
0925 02:56:44.369767 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.369804 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.370902 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.371381 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.371387 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.371672 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:44.372143 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.372157 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.373119 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.373122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.373934 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.373942 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.374479 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.374484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.374644 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.375074 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:56:44.375088 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:44.376495 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:44.377949 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:44.377954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:44.377966 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:44.378205 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:44.378494 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:56:44.378498 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:56:44.378499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:56:44.378638 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:56:44.379000 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:44.379006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:44.379208 (+ 202us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.379374 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.379641 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":99,"threads_started":1}
I20260925 02:56:44.381343 584 heartbeater.cc:347] Connected to a master server at 127.12.222.254:33001
I20260925 02:56:44.381575 584 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:44.382017 584 heartbeater.cc:511] Master 127.12.222.254:33001 requested a full tablet report, sending...
I20260925 02:56:44.382921 32586 ts_manager.cc:194] Registered new tserver with Master: 139af6d18f1c419691e05bf40beb3526 (127.12.222.195:37981)
I20260925 02:56:44.383419 32586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.222.195@KRBTEST.COM'} at 127.12.222.195:35877
I20260925 02:56:44.385596 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:56:44.393343 32627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.387425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.12.222.254:33001)
0925 02:56:44.387563 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.387566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.387705 (+ 139us) server_negotiation.cc:407] Connection header received
0925 02:56:44.387791 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.387793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.387825 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.387877 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:44.388489 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.388905 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.389587 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.389762 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.389825 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.390898 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:44.390901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:44.390903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:44.390922 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:44.392020 (+ 1098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:44.392425 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:44.392427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:44.392428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:44.392496 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:44.392696 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:44.392698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:44.392698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:44.392858 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:44.392950 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.393093 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.393179 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:56:44.395578 32586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:56:44.398337 32586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:56:44.411830 595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.406073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:37981 (local address 127.0.0.1:52776)
0925 02:56:44.406580 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.406622 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.406728 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.407753 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.407758 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.407785 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:44.408099 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.408105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.409386 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.409390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.410556 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.410564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.411294 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.411297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.411455 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.411492 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.411607 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.411711 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":242,"spinlock_wait_cycles":2944,"thread_start_us":90,"threads_started":1}
I20260925 02:56:44.411834 596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.406129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.12.222.195:37981)
0925 02:56:44.407338 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.407350 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.407370 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:56:44.407441 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.407446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.407616 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.407765 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:44.408185 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.409245 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.410661 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.411178 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.411305 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.411427 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.411635 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.411719 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1115,"thread_start_us":115,"threads_started":1}
I20260925 02:56:44.411978 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.406531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:34739 (local address 127.0.0.1:47430)
0925 02:56:44.406739 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.406750 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.406818 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.407758 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.407761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.407790 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:44.407977 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.407987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.409540 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.409543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.410610 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.410619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.411644 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.411647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.411733 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.411754 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.411798 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.411856 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":63,"spinlock_wait_cycles":4864,"thread_start_us":93,"threads_started":1}
I20260925 02:56:44.412525 598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.407104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47430 (local address 127.12.222.194:34739)
0925 02:56:44.407376 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.407385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.407413 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:56:44.407470 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.407473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.407597 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.408254 (+ 657us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:44.408418 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.409399 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.410736 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.411527 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.412091 (+ 564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.412206 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.412280 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.412381 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
I20260925 02:56:44.412806 599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.407802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:41347 (local address 127.0.0.1:57266)
0925 02:56:44.408023 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.408032 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.408120 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.408879 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.408881 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.408897 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:44.408974 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.408979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.410377 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.410379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.411393 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.411404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.412459 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.412462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.412547 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.412568 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.412646 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.412699 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260925 02:56:44.412906 600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.408218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57266 (local address 127.12.222.193:41347)
0925 02:56:44.408570 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.408578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.408594 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:56:44.408649 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.408652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.408761 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.408908 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:44.409056 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.410227 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.411531 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.412325 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.412468 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.412573 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.412674 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.412767 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":128,"threads_started":1}
I20260925 02:56:44.414891 517 tablet_service.cc:1467] Processing CreateTablet for tablet 55579c99b50f42bda4f8797345fb5166 (DEFAULT_TABLE table=test-table [id=4256501132cb4add8f4225bdc0c2ba26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:44.415443 32714 tablet_service.cc:1467] Processing CreateTablet for tablet 55579c99b50f42bda4f8797345fb5166 (DEFAULT_TABLE table=test-table [id=4256501132cb4add8f4225bdc0c2ba26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:44.415868 517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55579c99b50f42bda4f8797345fb5166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:44.416361 32714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55579c99b50f42bda4f8797345fb5166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:44.422137 382 tablet_service.cc:1467] Processing CreateTablet for tablet 55579c99b50f42bda4f8797345fb5166 (DEFAULT_TABLE table=test-table [id=4256501132cb4add8f4225bdc0c2ba26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:44.422720 601 tablet_bootstrap.cc:492] T 55579c99b50f42bda4f8797345fb5166 P a6d40d6338af44389609dc4d7e5edf71: Bootstrap starting.
I20260925 02:56:44.422984 382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55579c99b50f42bda4f8797345fb5166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:44.424526 602 tablet_bootstrap.cc:492] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526: Bootstrap starting.
I20260925 02:56:44.424952 601 tablet_bootstrap.cc:654] T 55579c99b50f42bda4f8797345fb5166 P a6d40d6338af44389609dc4d7e5edf71: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:44.426023 602 tablet_bootstrap.cc:654] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:44.426149 601 log.cc:824] T 55579c99b50f42bda4f8797345fb5166 P a6d40d6338af44389609dc4d7e5edf71: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:44.427033 602 log.cc:824] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:44.428292 601 tablet_bootstrap.cc:492] T 55579c99b50f42bda4f8797345fb5166 P a6d40d6338af44389609dc4d7e5edf71: No bootstrap required, opened a new log
I20260925 02:56:44.428478 601 ts_tablet_manager.cc:1397] T 55579c99b50f42bda4f8797345fb5166 P a6d40d6338af44389609dc4d7e5edf71: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 02:56:44.428603 602 tablet_bootstrap.cc:492] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526: No bootstrap required, opened a new log
I20260925 02:56:44.428787 602 ts_tablet_manager.cc:1397] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260925 02:56:44.429508 605 tablet_bootstrap.cc:492] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf: Bootstrap starting.
I20260925 02:56:44.431495 605 tablet_bootstrap.cc:654] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:44.431484 601 raft_consensus.cc:348] T 55579c99b50f42bda4f8797345fb5166 P a6d40d6338af44389609dc4d7e5edf71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139af6d18f1c419691e05bf40beb3526" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 37981 } } peers { permanent_uuid: "fd9ad1b544154db699127c7a1b9ca1bf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34739 } } peers { permanent_uuid: "a6d40d6338af44389609dc4d7e5edf71" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41347 } }
I20260925 02:56:44.431752 601 raft_consensus.cc:374] T 55579c99b50f42bda4f8797345fb5166 P a6d40d6338af44389609dc4d7e5edf71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:44.431818 601 raft_consensus.cc:729] T 55579c99b50f42bda4f8797345fb5166 P a6d40d6338af44389609dc4d7e5edf71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6d40d6338af44389609dc4d7e5edf71, State: Initialized, Role: FOLLOWER
I20260925 02:56:44.432277 601 consensus_queue.cc:260] T 55579c99b50f42bda4f8797345fb5166 P a6d40d6338af44389609dc4d7e5edf71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139af6d18f1c419691e05bf40beb3526" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 37981 } } peers { permanent_uuid: "fd9ad1b544154db699127c7a1b9ca1bf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34739 } } peers { permanent_uuid: "a6d40d6338af44389609dc4d7e5edf71" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41347 } }
I20260925 02:56:44.432468 602 raft_consensus.cc:348] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139af6d18f1c419691e05bf40beb3526" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 37981 } } peers { permanent_uuid: "fd9ad1b544154db699127c7a1b9ca1bf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34739 } } peers { permanent_uuid: "a6d40d6338af44389609dc4d7e5edf71" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41347 } }
I20260925 02:56:44.432725 602 raft_consensus.cc:374] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:44.432706 605 log.cc:824] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:44.432803 602 raft_consensus.cc:729] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139af6d18f1c419691e05bf40beb3526, State: Initialized, Role: FOLLOWER
I20260925 02:56:44.433053 314 heartbeater.cc:503] Master 127.12.222.254:33001 was elected leader, sending a full tablet report...
I20260925 02:56:44.433250 601 ts_tablet_manager.cc:1428] T 55579c99b50f42bda4f8797345fb5166 P a6d40d6338af44389609dc4d7e5edf71: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:56:44.433248 602 consensus_queue.cc:260] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139af6d18f1c419691e05bf40beb3526" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 37981 } } peers { permanent_uuid: "fd9ad1b544154db699127c7a1b9ca1bf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34739 } } peers { permanent_uuid: "a6d40d6338af44389609dc4d7e5edf71" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41347 } }
I20260925 02:56:44.434137 602 ts_tablet_manager.cc:1428] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 02:56:44.434406 584 heartbeater.cc:503] Master 127.12.222.254:33001 was elected leader, sending a full tablet report...
I20260925 02:56:44.434867 605 tablet_bootstrap.cc:492] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf: No bootstrap required, opened a new log
I20260925 02:56:44.435079 605 ts_tablet_manager.cc:1397] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:56:44.438689 605 raft_consensus.cc:348] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139af6d18f1c419691e05bf40beb3526" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 37981 } } peers { permanent_uuid: "fd9ad1b544154db699127c7a1b9ca1bf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34739 } } peers { permanent_uuid: "a6d40d6338af44389609dc4d7e5edf71" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41347 } }
I20260925 02:56:44.438925 605 raft_consensus.cc:374] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:44.438961 605 raft_consensus.cc:729] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9ad1b544154db699127c7a1b9ca1bf, State: Initialized, Role: FOLLOWER
I20260925 02:56:44.439309 605 consensus_queue.cc:260] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139af6d18f1c419691e05bf40beb3526" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 37981 } } peers { permanent_uuid: "fd9ad1b544154db699127c7a1b9ca1bf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34739 } } peers { permanent_uuid: "a6d40d6338af44389609dc4d7e5edf71" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41347 } }
I20260925 02:56:44.439848 449 heartbeater.cc:503] Master 127.12.222.254:33001 was elected leader, sending a full tablet report...
I20260925 02:56:44.440069 605 ts_tablet_manager.cc:1428] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260925 02:56:44.487890 315 tablet.cc:2369] T 55579c99b50f42bda4f8797345fb5166 P a6d40d6338af44389609dc4d7e5edf71: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:56:44.618327 585 tablet.cc:2369] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:56:44.674121 450 tablet.cc:2369] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:44.695298 608 raft_consensus.cc:482] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:56:44.695533 608 raft_consensus.cc:504] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139af6d18f1c419691e05bf40beb3526" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 37981 } } peers { permanent_uuid: "fd9ad1b544154db699127c7a1b9ca1bf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34739 } } peers { permanent_uuid: "a6d40d6338af44389609dc4d7e5edf71" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41347 } }
I20260925 02:56:44.696686 608 leader_election.cc:288] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd9ad1b544154db699127c7a1b9ca1bf (127.12.222.194:34739), a6d40d6338af44389609dc4d7e5edf71 (127.12.222.193:41347)
I20260925 02:56:44.701231 600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.697002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:53421 (local address 127.12.222.193:41347)
0925 02:56:44.697127 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.697132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.697147 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:56:44.697286 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.697289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.697336 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.697431 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:44.697695 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.698411 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.699997 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.700590 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.700701 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.700748 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.701044 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.701112 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:56:44.701231 598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.696989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:59133 (local address 127.12.222.194:34739)
0925 02:56:44.697152 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.697156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.697424 (+ 268us) server_negotiation.cc:407] Connection header received
0925 02:56:44.697545 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.697548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.697603 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.697688 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:44.697912 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.698531 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.699997 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.700590 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.700701 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.700748 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.701044 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.701110 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 02:56:44.701258 586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.696894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:41347 (local address 127.12.222.195:53421)
0925 02:56:44.697082 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.697096 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.697179 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.697460 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.697465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.697483 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:44.697596 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.697600 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.698513 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.698516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.699858 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.699869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.700706 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.700710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.700890 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.700921 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.701009 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.701110 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":127}
I20260925 02:56:44.701265 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.696900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:34739 (local address 127.12.222.195:59133)
0925 02:56:44.697328 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.697338 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:44.697444 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:44.697716 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:44.697718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:44.697732 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:44.697811 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.697815 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.698622 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.698624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.699858 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:44.699869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.700706 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.700710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:44.700913 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.700929 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:56:44.701009 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 02:56:44.701110 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":186,"thread_start_us":117,"threads_started":1}
I20260925 02:56:44.701996 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55579c99b50f42bda4f8797345fb5166" candidate_uuid: "139af6d18f1c419691e05bf40beb3526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d40d6338af44389609dc4d7e5edf71" is_pre_election: true
I20260925 02:56:44.702209 403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55579c99b50f42bda4f8797345fb5166" candidate_uuid: "139af6d18f1c419691e05bf40beb3526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd9ad1b544154db699127c7a1b9ca1bf" is_pre_election: true
I20260925 02:56:44.702284 32735 raft_consensus.cc:2454] T 55579c99b50f42bda4f8797345fb5166 P a6d40d6338af44389609dc4d7e5edf71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139af6d18f1c419691e05bf40beb3526 in term 0.
I20260925 02:56:44.702452 403 raft_consensus.cc:2454] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139af6d18f1c419691e05bf40beb3526 in term 0.
I20260925 02:56:44.702800 472 leader_election.cc:302] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139af6d18f1c419691e05bf40beb3526, a6d40d6338af44389609dc4d7e5edf71; no voters:
I20260925 02:56:44.703027 608 raft_consensus.cc:2781] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:56:44.703109 608 raft_consensus.cc:482] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:56:44.703137 608 raft_consensus.cc:3037] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:44.704155 608 raft_consensus.cc:504] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139af6d18f1c419691e05bf40beb3526" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 37981 } } peers { permanent_uuid: "fd9ad1b544154db699127c7a1b9ca1bf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34739 } } peers { permanent_uuid: "a6d40d6338af44389609dc4d7e5edf71" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41347 } }
I20260925 02:56:44.704557 608 leader_election.cc:288] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [CANDIDATE]: Term 1 election: Requested vote from peers fd9ad1b544154db699127c7a1b9ca1bf (127.12.222.194:34739), a6d40d6338af44389609dc4d7e5edf71 (127.12.222.193:41347)
I20260925 02:56:44.704874 403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55579c99b50f42bda4f8797345fb5166" candidate_uuid: "139af6d18f1c419691e05bf40beb3526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd9ad1b544154db699127c7a1b9ca1bf"
I20260925 02:56:44.705003 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55579c99b50f42bda4f8797345fb5166" candidate_uuid: "139af6d18f1c419691e05bf40beb3526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d40d6338af44389609dc4d7e5edf71"
I20260925 02:56:44.705052 403 raft_consensus.cc:3037] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:44.705144 32735 raft_consensus.cc:3037] T 55579c99b50f42bda4f8797345fb5166 P a6d40d6338af44389609dc4d7e5edf71 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:44.706015 403 raft_consensus.cc:2454] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139af6d18f1c419691e05bf40beb3526 in term 1.
I20260925 02:56:44.706137 32735 raft_consensus.cc:2454] T 55579c99b50f42bda4f8797345fb5166 P a6d40d6338af44389609dc4d7e5edf71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139af6d18f1c419691e05bf40beb3526 in term 1.
I20260925 02:56:44.706296 471 leader_election.cc:302] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139af6d18f1c419691e05bf40beb3526, fd9ad1b544154db699127c7a1b9ca1bf; no voters:
I20260925 02:56:44.706435 608 raft_consensus.cc:2781] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:44.706626 608 raft_consensus.cc:686] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [term 1 LEADER]: Becoming Leader. State: Replica: 139af6d18f1c419691e05bf40beb3526, State: Running, Role: LEADER
I20260925 02:56:44.706962 608 consensus_queue.cc:237] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139af6d18f1c419691e05bf40beb3526" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 37981 } } peers { permanent_uuid: "fd9ad1b544154db699127c7a1b9ca1bf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34739 } } peers { permanent_uuid: "a6d40d6338af44389609dc4d7e5edf71" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41347 } }
I20260925 02:56:44.709923 32585 catalog_manager.cc:5495] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 reported cstate change: term changed from 0 to 1, leader changed from <none> to 139af6d18f1c419691e05bf40beb3526 (127.12.222.195). New cstate: current_term: 1 leader_uuid: "139af6d18f1c419691e05bf40beb3526" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139af6d18f1c419691e05bf40beb3526" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 37981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd9ad1b544154db699127c7a1b9ca1bf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6d40d6338af44389609dc4d7e5edf71" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41347 } health_report { overall_health: UNKNOWN } } }
I20260925 02:56:44.768709 596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.765502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.12.222.195:37981)
0925 02:56:44.765627 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:44.765644 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:44.765676 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:56:44.765859 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:44.765861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:44.765913 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:44.766017 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:44.766247 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.766708 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.767452 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:44.767638 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:44.767734 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:44.768007 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:44.768349 (+ 342us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:44.768469 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:44.768538 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 02:56:44.768587 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:56:44.773635 32735 raft_consensus.cc:1261] T 55579c99b50f42bda4f8797345fb5166 P a6d40d6338af44389609dc4d7e5edf71 [term 1 FOLLOWER]: Refusing update from remote peer 139af6d18f1c419691e05bf40beb3526: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:44.773636 403 raft_consensus.cc:1261] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf [term 1 FOLLOWER]: Refusing update from remote peer 139af6d18f1c419691e05bf40beb3526: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:44.774283 608 consensus_queue.cc:1035] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6d40d6338af44389609dc4d7e5edf71" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:44.774451 611 consensus_queue.cc:1035] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd9ad1b544154db699127c7a1b9ca1bf" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 34739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:44.780069 614 mvcc.cc:205] Tried to move back new op lower bound from 7333089299546701824 to 7333089299282042880. Current Snapshot: MvccSnapshot[applied={T|T < 7333089299546701824}]
I20260925 02:56:44.781783 616 mvcc.cc:205] Tried to move back new op lower bound from 7333089299546701824 to 7333089299282042880. Current Snapshot: MvccSnapshot[applied={T|T < 7333089299546701824}]
I20260925 02:56:44.782120 615 mvcc.cc:205] Tried to move back new op lower bound from 7333089299546701824 to 7333089299282042880. Current Snapshot: MvccSnapshot[applied={T|T < 7333089299546701824}]
I20260925 02:56:44.789103 32585 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:44.789213 32585 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:44.791287 32585 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f525a1e0c8ef48b9887b9f073c59609d: Sending DeleteTablet for 3 replicas of tablet 55579c99b50f42bda4f8797345fb5166
I20260925 02:56:44.791936 382 tablet_service.cc:1514] Processing DeleteTablet for tablet 55579c99b50f42bda4f8797345fb5166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:44 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:47430
I20260925 02:56:44.791936 517 tablet_service.cc:1514] Processing DeleteTablet for tablet 55579c99b50f42bda4f8797345fb5166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:44 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:52776
I20260925 02:56:44.792084 32714 tablet_service.cc:1514] Processing DeleteTablet for tablet 55579c99b50f42bda4f8797345fb5166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:44 UTC) from {username='kudu', principal='kudu/127.12.222.254@KRBTEST.COM'} at 127.0.0.1:57266
I20260925 02:56:44.792408 623 tablet_replica.cc:330] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf: stopping tablet replica
I20260925 02:56:44.792508 625 tablet_replica.cc:330] T 55579c99b50f42bda4f8797345fb5166 P a6d40d6338af44389609dc4d7e5edf71: stopping tablet replica
I20260925 02:56:44.792753 623 raft_consensus.cc:2229] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:44.792753 625 raft_consensus.cc:2229] T 55579c99b50f42bda4f8797345fb5166 P a6d40d6338af44389609dc4d7e5edf71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:44.793012 625 raft_consensus.cc:2258] T 55579c99b50f42bda4f8797345fb5166 P a6d40d6338af44389609dc4d7e5edf71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:44.793030 623 raft_consensus.cc:2258] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:44.793190 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 32650
I20260925 02:56:44.793953 624 tablet_replica.cc:330] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526: stopping tablet replica
I20260925 02:56:44.794212 624 raft_consensus.cc:2229] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:56:44.794528 624 raft_consensus.cc:2258] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:44.795907 623 ts_tablet_manager.cc:1905] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:44.795907 624 ts_tablet_manager.cc:1905] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:44.798516 623 ts_tablet_manager.cc:1918] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:56:44.798527 624 ts_tablet_manager.cc:1918] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:56:44.798622 623 log.cc:1196] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/55579c99b50f42bda4f8797345fb5166
I20260925 02:56:44.798732 624 log.cc:1196] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/55579c99b50f42bda4f8797345fb5166
I20260925 02:56:44.798964 623 ts_tablet_manager.cc:1939] T 55579c99b50f42bda4f8797345fb5166 P fd9ad1b544154db699127c7a1b9ca1bf: Deleting consensus metadata
I20260925 02:56:44.799171 624 ts_tablet_manager.cc:1939] T 55579c99b50f42bda4f8797345fb5166 P 139af6d18f1c419691e05bf40beb3526: Deleting consensus metadata
I20260925 02:56:44.800351 32572 catalog_manager.cc:4841] TS 139af6d18f1c419691e05bf40beb3526 (127.12.222.195:37981): tablet 55579c99b50f42bda4f8797345fb5166 (table test-table [id=4256501132cb4add8f4225bdc0c2ba26]) successfully deleted
I20260925 02:56:44.800351 32570 catalog_manager.cc:4841] TS fd9ad1b544154db699127c7a1b9ca1bf (127.12.222.194:34739): tablet 55579c99b50f42bda4f8797345fb5166 (table test-table [id=4256501132cb4add8f4225bdc0c2ba26]) successfully deleted
W20260925 02:56:44.803519 32571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.222.193:41347 (error 108)
I20260925 02:56:44.804333 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 319
I20260925 02:56:44.804507 595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:44.804189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:41347 (local address 127.0.0.1:57272)
0925 02:56:44.804364 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:44.804433 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.193:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260925 02:56:44.805090 32571 catalog_manager.cc:4584] TS a6d40d6338af44389609dc4d7e5edf71 (127.12.222.193:41347): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55579c99b50f42bda4f8797345fb5166: Network error: Client connection negotiation failed: client connection to 127.12.222.193:41347: connect: Connection refused (error 111)
I20260925 02:56:44.810689 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 454
I20260925 02:56:44.816164 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 32553
2026-09-25T02:56:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2298 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T02:56:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:44Z Disabled control of system clock
I20260925 02:56:44.845237 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:42809
--webserver_interface=127.12.222.254
--webserver_port=0
--builtin_ntp_servers=127.12.222.212:43603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:42809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:56:44.945959 630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:44.946144 630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:44.946190 630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:44.949647 630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:44.949687 630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:44.949700 630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:44.949715 630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:44.949728 630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:44.954042 630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:42809
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:42809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:44.954782 630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:44.955852 630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:44.963024 635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:44.963043 630 server_base.cc:1034] running on GCE node
W20260925 02:56:44.962967 636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:44.962990 638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:44.964056 630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:44.965015 630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:44.966172 630 hybrid_clock.cc:648] HybridClock initialized: now 1790305004966155 us; error 26 us; skew 500 ppm
I20260925 02:56:44.967774 630 webserver.cc:460] Webserver started at http://127.12.222.254:33415/ using document root <none> and password file <none>
I20260925 02:56:44.968312 630 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:44.968374 630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:44.968583 630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:44.970270 630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "c9757684f8e14d64949daf6a5f100a01"
format_stamp: "Formatted at 2026-09-25 02:56:44 on dist-test-slave-252r"
server_key: "f5e904904c7cf7704be614b3ba47ab63"
server_key_iv: "2d6ca25c59149b40e65a606b0da09a4b"
server_key_version: "encryptionkey@0"
I20260925 02:56:44.970785 630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "c9757684f8e14d64949daf6a5f100a01"
format_stamp: "Formatted at 2026-09-25 02:56:44 on dist-test-slave-252r"
server_key: "f5e904904c7cf7704be614b3ba47ab63"
server_key_iv: "2d6ca25c59149b40e65a606b0da09a4b"
server_key_version: "encryptionkey@0"
I20260925 02:56:44.973881 630 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:56:44.975986 644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:44.977105 630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:44.977221 630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "c9757684f8e14d64949daf6a5f100a01"
format_stamp: "Formatted at 2026-09-25 02:56:44 on dist-test-slave-252r"
server_key: "f5e904904c7cf7704be614b3ba47ab63"
server_key_iv: "2d6ca25c59149b40e65a606b0da09a4b"
server_key_version: "encryptionkey@0"
I20260925 02:56:44.977316 630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:44.992455 630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:44.992981 630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:44.993148 630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:44.999013 630 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:42809
I20260925 02:56:44.999044 696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:42809 every 8 connection(s)
I20260925 02:56:44.999789 630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:45.000036 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 630
I20260925 02:56:45.000154 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:56:45.000499 13179 external_mini_cluster.cc:1428] Setting key dfc32eba6656dd5a61cc3e99906d8149
I20260925 02:56:45.003185 697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:45.009282 697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01: Bootstrap starting.
I20260925 02:56:45.010021 700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.001806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46540 (local address 127.12.222.254:42809)
0925 02:56:45.002415 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:45.002434 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:45.002473 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:56:45.003239 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:45.003261 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:45.003707 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:45.004052 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:45.004481 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.005867 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.006661 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.007284 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.007446 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.008192 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:45.008198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:45.008212 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:45.008690 (+ 478us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:45.008751 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:45.008776 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:45.008872 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:45.009000 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 02:56:45.009231 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":105,"threads_started":1}
I20260925 02:56:45.011583 697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:45.012377 697 log.cc:824] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:45.014071 697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01: No bootstrap required, opened a new log
I20260925 02:56:45.016956 697 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9757684f8e14d64949daf6a5f100a01" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42809 } }
I20260925 02:56:45.017131 697 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:45.017163 697 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9757684f8e14d64949daf6a5f100a01, State: Initialized, Role: FOLLOWER
I20260925 02:56:45.017501 697 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9757684f8e14d64949daf6a5f100a01" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42809 } }
I20260925 02:56:45.017611 697 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:45.017653 697 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:45.017709 697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:45.018405 697 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9757684f8e14d64949daf6a5f100a01" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42809 } }
I20260925 02:56:45.018671 697 leader_election.cc:302] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9757684f8e14d64949daf6a5f100a01; no voters:
I20260925 02:56:45.018875 697 leader_election.cc:288] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:45.019007 703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:45.019235 703 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01 [term 1 LEADER]: Becoming Leader. State: Replica: c9757684f8e14d64949daf6a5f100a01, State: Running, Role: LEADER
I20260925 02:56:45.019510 703 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9757684f8e14d64949daf6a5f100a01" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42809 } }
I20260925 02:56:45.019946 697 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:45.021095 705 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9757684f8e14d64949daf6a5f100a01. Latest consensus state: current_term: 1 leader_uuid: "c9757684f8e14d64949daf6a5f100a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9757684f8e14d64949daf6a5f100a01" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42809 } } }
I20260925 02:56:45.021101 704 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9757684f8e14d64949daf6a5f100a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9757684f8e14d64949daf6a5f100a01" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 42809 } } }
I20260925 02:56:45.021245 705 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:45.021247 704 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:45.021607 712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:45.023963 712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:45.028733 712 catalog_manager.cc:1347] Generated new cluster ID: 21f36c946d974e758b8815a23b1ba092
I20260925 02:56:45.028786 712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:45.041446 712 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:45.042209 712 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:45.045712 712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01: Generated new TSK 0
I20260925 02:56:45.046228 712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:45.051829 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.193:0
--local_ip_for_outbound_sockets=127.12.222.193
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:42809
--builtin_ntp_servers=127.12.222.212:43603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:56:45.149653 722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:45.149839 722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:45.149870 722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:45.153607 722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:45.153645 722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:45.153740 722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.193
I20260925 02:56:45.158121 722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.222.193
--webserver_port=0
--tserver_master_addrs=127.12.222.254:42809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.193
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:45.158929 722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:45.160045 722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:45.168282 730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:45.168303 728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:45.168321 727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:45.168457 722 server_base.cc:1034] running on GCE node
I20260925 02:56:45.168882 722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:45.169385 722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:45.170559 722 hybrid_clock.cc:648] HybridClock initialized: now 1790305005170532 us; error 36 us; skew 500 ppm
I20260925 02:56:45.172004 722 webserver.cc:460] Webserver started at http://127.12.222.193:45465/ using document root <none> and password file <none>
I20260925 02:56:45.172540 722 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:45.172618 722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:45.172812 722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:45.174561 722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-0/data/instance:
uuid: "804ab5a6027147b58c3017dba58648ff"
format_stamp: "Formatted at 2026-09-25 02:56:45 on dist-test-slave-252r"
server_key: "484d88ab91d5615c23754c98033a0cf1"
server_key_iv: "4be06122c78ffecbd2beb51dbcd45829"
server_key_version: "encryptionkey@0"
I20260925 02:56:45.175113 722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance:
uuid: "804ab5a6027147b58c3017dba58648ff"
format_stamp: "Formatted at 2026-09-25 02:56:45 on dist-test-slave-252r"
server_key: "484d88ab91d5615c23754c98033a0cf1"
server_key_iv: "4be06122c78ffecbd2beb51dbcd45829"
server_key_version: "encryptionkey@0"
I20260925 02:56:45.178256 722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 02:56:45.180436 736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:45.181532 722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:56:45.181665 722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-0/wal
uuid: "804ab5a6027147b58c3017dba58648ff"
format_stamp: "Formatted at 2026-09-25 02:56:45 on dist-test-slave-252r"
server_key: "484d88ab91d5615c23754c98033a0cf1"
server_key_iv: "4be06122c78ffecbd2beb51dbcd45829"
server_key_version: "encryptionkey@0"
I20260925 02:56:45.181766 722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:45.197906 722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:45.198413 722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:45.198572 722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:45.199047 722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:45.199812 722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:45.199874 722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:45.199955 722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:45.200001 722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:45.207808 722 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.193:41637
I20260925 02:56:45.207904 849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.193:41637 every 8 connection(s)
I20260925 02:56:45.208618 722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-0/data/info.pb
I20260925 02:56:45.216162 700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.210190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.193:60999 (local address 127.12.222.254:42809)
0925 02:56:45.210313 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:45.210316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:45.211168 (+ 852us) server_negotiation.cc:407] Connection header received
0925 02:56:45.212326 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:45.212328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:45.212374 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:45.212442 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:45.213123 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.213584 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.214591 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.214792 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.214864 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.215793 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:45.215795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:45.215797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:45.215807 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:45.215820 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:45.215826 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:45.215874 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:45.216004 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 02:56:45.216062 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:56:45.216822 852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.210508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42809 (local address 127.12.222.193:60999)
0925 02:56:45.211042 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:45.211075 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:45.212109 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:45.212572 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:45.212577 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:45.212645 (+ 68us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:45.213015 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.213026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.213683 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.213685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.214490 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.214498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.214915 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.214918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.215047 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.215589 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:56:45.215590 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:45.215622 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:56:45.215674 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:45.215899 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:45.215905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:45.215909 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:56:45.215990 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 02:56:45.216164 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":97,"threads_started":1}
I20260925 02:56:45.217425 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 722
I20260925 02:56:45.217571 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-0/wal/instance
I20260925 02:56:45.217854 13179 external_mini_cluster.cc:1428] Setting key 6267a281bbff4b76095f66b2291026db
I20260925 02:56:45.217943 850 heartbeater.cc:347] Connected to a master server at 127.12.222.254:42809
I20260925 02:56:45.218155 850 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:45.218577 850 heartbeater.cc:511] Master 127.12.222.254:42809 requested a full tablet report, sending...
I20260925 02:56:45.219682 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.194:0
--local_ip_for_outbound_sockets=127.12.222.194
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:42809
--builtin_ntp_servers=127.12.222.212:43603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:56:45.219980 661 ts_manager.cc:194] Registered new tserver with Master: 804ab5a6027147b58c3017dba58648ff (127.12.222.193:41637)
I20260925 02:56:45.221449 661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.222.193:60999
W20260925 02:56:45.320670 853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:45.320912 853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:45.320955 853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:45.324898 853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:45.324941 853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:45.324990 853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.194
I20260925 02:56:45.329181 853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.222.194
--webserver_port=0
--tserver_master_addrs=127.12.222.254:42809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.194
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:45.329932 853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:45.331051 853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:56:45.339650 853 server_base.cc:1034] running on GCE node
W20260925 02:56:45.339557 861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:45.339557 859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:45.339560 858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:45.340157 853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:45.340706 853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:45.341904 853 hybrid_clock.cc:648] HybridClock initialized: now 1790305005341890 us; error 33 us; skew 500 ppm
I20260925 02:56:45.343497 853 webserver.cc:460] Webserver started at http://127.12.222.194:37601/ using document root <none> and password file <none>
I20260925 02:56:45.344045 853 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:45.344135 853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:45.344390 853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:45.346187 853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-1/data/instance:
uuid: "4431abb4fc45461c88eceb626631b19d"
format_stamp: "Formatted at 2026-09-25 02:56:45 on dist-test-slave-252r"
server_key: "3d571e3a0baf541d06594cafac944ad9"
server_key_iv: "47c252b273ea49e87cdc16c588cc0ad9"
server_key_version: "encryptionkey@0"
I20260925 02:56:45.346681 853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance:
uuid: "4431abb4fc45461c88eceb626631b19d"
format_stamp: "Formatted at 2026-09-25 02:56:45 on dist-test-slave-252r"
server_key: "3d571e3a0baf541d06594cafac944ad9"
server_key_iv: "47c252b273ea49e87cdc16c588cc0ad9"
server_key_version: "encryptionkey@0"
I20260925 02:56:45.349915 853 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:56:45.352032 867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:45.353094 853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:45.353214 853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-1/wal
uuid: "4431abb4fc45461c88eceb626631b19d"
format_stamp: "Formatted at 2026-09-25 02:56:45 on dist-test-slave-252r"
server_key: "3d571e3a0baf541d06594cafac944ad9"
server_key_iv: "47c252b273ea49e87cdc16c588cc0ad9"
server_key_version: "encryptionkey@0"
I20260925 02:56:45.353309 853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:45.363129 853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:45.363669 853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:45.363853 853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:45.364439 853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:45.365269 853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:45.365339 853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:45.365398 853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:45.365446 853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:45.372750 853 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.194:37215
I20260925 02:56:45.372819 983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.194:37215 every 8 connection(s)
I20260925 02:56:45.373481 853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-1/data/info.pb
I20260925 02:56:45.376080 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 853
I20260925 02:56:45.376200 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-1/wal/instance
I20260925 02:56:45.376503 13179 external_mini_cluster.cc:1428] Setting key 177d341021857e372c73668586be60f3
I20260925 02:56:45.378440 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.222.195:0
--local_ip_for_outbound_sockets=127.12.222.195
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:42809
--builtin_ntp_servers=127.12.222.212:43603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:56:45.382707 700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.375016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.194:55181 (local address 127.12.222.254:42809)
0925 02:56:45.375152 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:45.375156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:45.376104 (+ 948us) server_negotiation.cc:407] Connection header received
0925 02:56:45.377659 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:45.377662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:45.377729 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:45.377808 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:45.378708 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.379531 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.380835 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.381090 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.381195 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.382301 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:45.382304 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:45.382306 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:45.382316 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:45.382332 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:45.382340 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:45.382400 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:45.382562 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 02:56:45.382609 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:56:45.383771 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.375366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42809 (local address 127.12.222.194:55181)
0925 02:56:45.375950 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:45.376001 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:45.377431 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:45.377998 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:45.378007 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:45.378094 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:45.378500 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.378511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.379650 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.379652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.380690 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.380700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.381243 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.381248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.381461 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.382036 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:56:45.382040 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:45.382086 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:56:45.382157 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:45.382420 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:45.382431 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:45.382439 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:56:45.382557 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 02:56:45.382818 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":96,"threads_started":1}
I20260925 02:56:45.384877 984 heartbeater.cc:347] Connected to a master server at 127.12.222.254:42809
I20260925 02:56:45.385128 984 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:45.385675 984 heartbeater.cc:511] Master 127.12.222.254:42809 requested a full tablet report, sending...
I20260925 02:56:45.386740 661 ts_manager.cc:194] Registered new tserver with Master: 4431abb4fc45461c88eceb626631b19d (127.12.222.194:37215)
I20260925 02:56:45.387524 661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.222.194:55181
W20260925 02:56:45.483067 987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:45.483255 987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:45.483286 987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:45.487100 987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:45.487141 987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:45.487186 987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.222.195
I20260925 02:56:45.491431 987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:43603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.222.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.222.195
--webserver_port=0
--tserver_master_addrs=127.12.222.254:42809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.222.195
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:45.492152 987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:45.493319 987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:45.501842 993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:45.501883 992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:45.501866 995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:45.502101 987 server_base.cc:1034] running on GCE node
I20260925 02:56:45.502491 987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:45.503023 987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:45.504196 987 hybrid_clock.cc:648] HybridClock initialized: now 1790305005504186 us; error 25 us; skew 500 ppm
I20260925 02:56:45.505757 987 webserver.cc:460] Webserver started at http://127.12.222.195:42635/ using document root <none> and password file <none>
I20260925 02:56:45.506255 987 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:45.506342 987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:45.506554 987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:45.508450 987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-2/data/instance:
uuid: "e223d64568524102ab1a3fc2bbd935b6"
format_stamp: "Formatted at 2026-09-25 02:56:45 on dist-test-slave-252r"
server_key: "5cbef738679778971b0ec5773b395d9e"
server_key_iv: "0ccec744dea80e16ddceb901469bf8b3"
server_key_version: "encryptionkey@0"
I20260925 02:56:45.508960 987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance:
uuid: "e223d64568524102ab1a3fc2bbd935b6"
format_stamp: "Formatted at 2026-09-25 02:56:45 on dist-test-slave-252r"
server_key: "5cbef738679778971b0ec5773b395d9e"
server_key_iv: "0ccec744dea80e16ddceb901469bf8b3"
server_key_version: "encryptionkey@0"
I20260925 02:56:45.512178 987 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:56:45.514361 1001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:45.515443 987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 02:56:45.515570 987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-2/wal
uuid: "e223d64568524102ab1a3fc2bbd935b6"
format_stamp: "Formatted at 2026-09-25 02:56:45 on dist-test-slave-252r"
server_key: "5cbef738679778971b0ec5773b395d9e"
server_key_iv: "0ccec744dea80e16ddceb901469bf8b3"
server_key_version: "encryptionkey@0"
I20260925 02:56:45.515689 987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:45.530869 987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:45.531406 987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:45.531576 987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:45.532060 987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:56:45.532930 987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:56:45.532996 987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:45.533056 987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:56:45.533104 987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:45.541231 987 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.195:45367
I20260925 02:56:45.541303 1115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.195:45367 every 8 connection(s)
I20260925 02:56:45.542032 987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-2/data/info.pb
I20260925 02:56:45.545161 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 987
I20260925 02:56:45.545287 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-2/wal/instance
I20260925 02:56:45.545599 13179 external_mini_cluster.cc:1428] Setting key 7694dd124dbd52bd3124ef5d111377b4
I20260925 02:56:45.551023 700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.543601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:39907 (local address 127.12.222.254:42809)
0925 02:56:45.543753 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:45.543758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:45.544758 (+ 1000us) server_negotiation.cc:407] Connection header received
0925 02:56:45.546470 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:45.546476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:45.546534 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:45.546612 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:45.547688 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.548317 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.549359 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.549564 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.549635 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.550656 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:45.550659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:45.550661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:45.550671 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:45.550685 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:45.550691 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:45.550743 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:45.550875 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 02:56:45.550919 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:56:45.551818 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.543991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:42809 (local address 127.12.222.195:39907)
0925 02:56:45.544625 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:45.544662 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:45.546215 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:45.546848 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:45.546859 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:45.546965 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:56:45.547544 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.547563 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.548440 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.548443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.549263 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.549271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.549684 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.549687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.549814 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.550413 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:56:45.550414 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:56:45.550468 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:56:45.550526 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:56:45.550761 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:56:45.550767 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:56:45.550771 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:56:45.550859 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 02:56:45.551083 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":109,"threads_started":1}
I20260925 02:56:45.552863 1116 heartbeater.cc:347] Connected to a master server at 127.12.222.254:42809
I20260925 02:56:45.553040 1116 heartbeater.cc:464] Registering TS with master...
I20260925 02:56:45.553457 1116 heartbeater.cc:511] Master 127.12.222.254:42809 requested a full tablet report, sending...
I20260925 02:56:45.554325 661 ts_manager.cc:194] Registered new tserver with Master: e223d64568524102ab1a3fc2bbd935b6 (127.12.222.195:45367)
I20260925 02:56:45.554863 661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.222.195:39907
I20260925 02:56:45.559264 13179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:56:45.564360 700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.561126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.12.222.254:42809)
0925 02:56:45.561260 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:45.561264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:45.561405 (+ 141us) server_negotiation.cc:407] Connection header received
0925 02:56:45.561487 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:45.561490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:45.561524 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:45.561583 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:45.561800 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.562282 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.563020 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.563210 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.563279 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.563958 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:45.563961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:45.563963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:45.563973 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:45.563989 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:45.563997 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:45.564053 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:45.564134 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 02:56:45.564172 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:56:45.566898 661 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:56:45.569664 661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:56:45.582139 1127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.577065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:37215 (local address 127.0.0.1:39588)
0925 02:56:45.577347 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:45.577387 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:45.577489 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:45.578338 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:45.578343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:45.578364 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:45.578487 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.578495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.579611 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.579617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.580653 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.580661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.581659 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.581663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.581762 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.581796 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 02:56:45.581900 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 02:56:45.582015 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260925 02:56:45.582155 1128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.577193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39588 (local address 127.12.222.194:37215)
0925 02:56:45.577672 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:45.577688 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:45.577714 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:56:45.577790 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:45.577795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:45.577992 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:45.578167 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:45.578673 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.579510 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.580783 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.581561 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.581676 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.581837 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:45.581923 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 02:56:45.582021 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":144,"threads_started":1}
I20260925 02:56:45.583428 1129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.577414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.195:45367 (local address 127.0.0.1:53172)
0925 02:56:45.578137 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:45.578151 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:45.578224 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:45.579239 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:45.579241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:45.579263 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:45.579362 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.579369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.580742 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.580744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.581891 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.581902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.583082 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.583085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.583167 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.583189 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:56:45.583249 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 02:56:45.583307 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":15,"thread_start_us":79,"threads_started":1}
I20260925 02:56:45.583493 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.577757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53172 (local address 127.12.222.195:45367)
0925 02:56:45.578853 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:45.578862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:45.578881 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:56:45.578942 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:45.578946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:45.579096 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:45.579512 (+ 416us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:45.579622 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.580623 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.582050 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.582958 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.583136 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.583253 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:45.583299 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 02:56:45.583357 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1011,"thread_start_us":139,"threads_started":1}
I20260925 02:56:45.584131 1130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.577696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:41637 (local address 127.0.0.1:36720)
0925 02:56:45.578505 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:45.578515 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:45.578612 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:45.579095 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:45.579097 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:45.579116 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:45.579193 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.579198 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.580663 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.580669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.581968 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.581976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.583006 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.583009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.583839 (+ 830us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.583866 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 02:56:45.583933 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 02:56:45.583996 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":2,"thread_start_us":76,"threads_started":1}
I20260925 02:56:45.584653 1132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.578086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36720 (local address 127.12.222.193:41637)
0925 02:56:45.578476 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:45.578488 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:45.578599 (+ 111us) server_negotiation.cc:407] Connection header received
0925 02:56:45.578750 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:45.578756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:45.578950 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:45.579091 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:45.579303 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.580430 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.582084 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.582902 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.583035 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.583190 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:45.584362 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0925 02:56:45.584504 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":145,"threads_started":1}
I20260925 02:56:45.585065 917 tablet_service.cc:1467] Processing CreateTablet for tablet 3065d8d260a44d7989867fb60d464d40 (DEFAULT_TABLE table=test-table [id=b67e254442354c80a600917de8a3a8f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:45.586091 917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3065d8d260a44d7989867fb60d464d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:45.585943 1049 tablet_service.cc:1467] Processing CreateTablet for tablet 3065d8d260a44d7989867fb60d464d40 (DEFAULT_TABLE table=test-table [id=b67e254442354c80a600917de8a3a8f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:45.586755 1049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3065d8d260a44d7989867fb60d464d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:45.591823 1133 tablet_bootstrap.cc:492] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d: Bootstrap starting.
I20260925 02:56:45.592937 1134 tablet_bootstrap.cc:492] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6: Bootstrap starting.
I20260925 02:56:45.594135 784 tablet_service.cc:1467] Processing CreateTablet for tablet 3065d8d260a44d7989867fb60d464d40 (DEFAULT_TABLE table=test-table [id=b67e254442354c80a600917de8a3a8f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:56:45.594587 1133 tablet_bootstrap.cc:654] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:45.594933 1134 tablet_bootstrap.cc:654] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:45.594961 784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3065d8d260a44d7989867fb60d464d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:45.595502 1133 log.cc:824] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:45.595685 1134 log.cc:824] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:45.597292 1133 tablet_bootstrap.cc:492] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d: No bootstrap required, opened a new log
I20260925 02:56:45.597503 1133 ts_tablet_manager.cc:1397] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:56:45.598047 1134 tablet_bootstrap.cc:492] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6: No bootstrap required, opened a new log
I20260925 02:56:45.598223 1134 ts_tablet_manager.cc:1397] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:56:45.600667 1133 raft_consensus.cc:348] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4431abb4fc45461c88eceb626631b19d" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 37215 } } peers { permanent_uuid: "804ab5a6027147b58c3017dba58648ff" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41637 } } peers { permanent_uuid: "e223d64568524102ab1a3fc2bbd935b6" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45367 } }
I20260925 02:56:45.600919 1133 raft_consensus.cc:374] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:45.600973 1133 raft_consensus.cc:729] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4431abb4fc45461c88eceb626631b19d, State: Initialized, Role: FOLLOWER
I20260925 02:56:45.601432 1133 consensus_queue.cc:260] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4431abb4fc45461c88eceb626631b19d" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 37215 } } peers { permanent_uuid: "804ab5a6027147b58c3017dba58648ff" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41637 } } peers { permanent_uuid: "e223d64568524102ab1a3fc2bbd935b6" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45367 } }
I20260925 02:56:45.601697 1137 tablet_bootstrap.cc:492] T 3065d8d260a44d7989867fb60d464d40 P 804ab5a6027147b58c3017dba58648ff: Bootstrap starting.
I20260925 02:56:45.601953 1134 raft_consensus.cc:348] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4431abb4fc45461c88eceb626631b19d" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 37215 } } peers { permanent_uuid: "804ab5a6027147b58c3017dba58648ff" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41637 } } peers { permanent_uuid: "e223d64568524102ab1a3fc2bbd935b6" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45367 } }
I20260925 02:56:45.602206 1134 raft_consensus.cc:374] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:45.602272 1134 raft_consensus.cc:729] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e223d64568524102ab1a3fc2bbd935b6, State: Initialized, Role: FOLLOWER
I20260925 02:56:45.602417 1133 ts_tablet_manager.cc:1428] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 02:56:45.602530 984 heartbeater.cc:503] Master 127.12.222.254:42809 was elected leader, sending a full tablet report...
I20260925 02:56:45.602679 1134 consensus_queue.cc:260] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4431abb4fc45461c88eceb626631b19d" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 37215 } } peers { permanent_uuid: "804ab5a6027147b58c3017dba58648ff" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41637 } } peers { permanent_uuid: "e223d64568524102ab1a3fc2bbd935b6" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45367 } }
I20260925 02:56:45.603260 1116 heartbeater.cc:503] Master 127.12.222.254:42809 was elected leader, sending a full tablet report...
I20260925 02:56:45.603570 1134 ts_tablet_manager.cc:1428] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260925 02:56:45.604044 1137 tablet_bootstrap.cc:654] T 3065d8d260a44d7989867fb60d464d40 P 804ab5a6027147b58c3017dba58648ff: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:45.604875 1137 log.cc:824] T 3065d8d260a44d7989867fb60d464d40 P 804ab5a6027147b58c3017dba58648ff: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:45.606822 1137 tablet_bootstrap.cc:492] T 3065d8d260a44d7989867fb60d464d40 P 804ab5a6027147b58c3017dba58648ff: No bootstrap required, opened a new log
I20260925 02:56:45.607009 1137 ts_tablet_manager.cc:1397] T 3065d8d260a44d7989867fb60d464d40 P 804ab5a6027147b58c3017dba58648ff: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260925 02:56:45.609843 1137 raft_consensus.cc:348] T 3065d8d260a44d7989867fb60d464d40 P 804ab5a6027147b58c3017dba58648ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4431abb4fc45461c88eceb626631b19d" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 37215 } } peers { permanent_uuid: "804ab5a6027147b58c3017dba58648ff" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41637 } } peers { permanent_uuid: "e223d64568524102ab1a3fc2bbd935b6" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45367 } }
I20260925 02:56:45.610013 1137 raft_consensus.cc:374] T 3065d8d260a44d7989867fb60d464d40 P 804ab5a6027147b58c3017dba58648ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:45.610085 1137 raft_consensus.cc:729] T 3065d8d260a44d7989867fb60d464d40 P 804ab5a6027147b58c3017dba58648ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804ab5a6027147b58c3017dba58648ff, State: Initialized, Role: FOLLOWER
I20260925 02:56:45.610406 1137 consensus_queue.cc:260] T 3065d8d260a44d7989867fb60d464d40 P 804ab5a6027147b58c3017dba58648ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4431abb4fc45461c88eceb626631b19d" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 37215 } } peers { permanent_uuid: "804ab5a6027147b58c3017dba58648ff" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41637 } } peers { permanent_uuid: "e223d64568524102ab1a3fc2bbd935b6" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45367 } }
I20260925 02:56:45.610960 850 heartbeater.cc:503] Master 127.12.222.254:42809 was elected leader, sending a full tablet report...
I20260925 02:56:45.611186 1137 ts_tablet_manager.cc:1428] T 3065d8d260a44d7989867fb60d464d40 P 804ab5a6027147b58c3017dba58648ff: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
W20260925 02:56:45.624416 985 tablet.cc:2369] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:56:45.709609 851 tablet.cc:2369] T 3065d8d260a44d7989867fb60d464d40 P 804ab5a6027147b58c3017dba58648ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:56:45.793036 1117 tablet.cc:2369] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:56:45.801622 1139 raft_consensus.cc:482] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:56:45.801815 1139 raft_consensus.cc:504] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4431abb4fc45461c88eceb626631b19d" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 37215 } } peers { permanent_uuid: "804ab5a6027147b58c3017dba58648ff" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41637 } } peers { permanent_uuid: "e223d64568524102ab1a3fc2bbd935b6" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45367 } }
I20260925 02:56:45.802973 1139 leader_election.cc:288] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4431abb4fc45461c88eceb626631b19d (127.12.222.194:37215), 804ab5a6027147b58c3017dba58648ff (127.12.222.193:41637)
I20260925 02:56:45.807291 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.803231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:37215 (local address 127.12.222.195:45713)
0925 02:56:45.803363 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:45.803377 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:45.803507 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:45.803973 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:45.803976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:45.803994 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:45.804152 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.804160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.805073 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.805077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.806139 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.806167 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.806877 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.806884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.806983 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.807011 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 02:56:45.807081 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 02:56:45.807142 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":45}
I20260925 02:56:45.807330 1128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.803350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:45713 (local address 127.12.222.194:37215)
0925 02:56:45.803497 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:45.803502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:45.803517 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:56:45.803804 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:45.803808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:45.803865 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:45.803950 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:45.804303 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.804962 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.806286 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.806753 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.806846 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.806882 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:45.807120 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 02:56:45.807207 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:56:45.807426 1132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.803658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.222.195:53757 (local address 127.12.222.193:41637)
0925 02:56:45.803801 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:45.803806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:45.804052 (+ 246us) server_negotiation.cc:407] Connection header received
0925 02:56:45.804173 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:45.804178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:45.804262 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:45.804334 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:56:45.804596 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.805143 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.806365 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.806848 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.806969 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.807014 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:45.807222 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 02:56:45.807290 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:56:45.807432 1142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.803541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:41637 (local address 127.12.222.195:53757)
0925 02:56:45.803964 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:45.803977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:56:45.804070 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:56:45.804347 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:56:45.804349 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:56:45.804367 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:56:45.804483 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.804490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.805252 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.805254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.806171 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:56:45.806183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.806993 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.806996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:56:45.807091 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.807117 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:56:45.807191 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:56:45.807250 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":151,"threads_started":1}
I20260925 02:56:45.808090 937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3065d8d260a44d7989867fb60d464d40" candidate_uuid: "e223d64568524102ab1a3fc2bbd935b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4431abb4fc45461c88eceb626631b19d" is_pre_election: true
I20260925 02:56:45.808077 804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3065d8d260a44d7989867fb60d464d40" candidate_uuid: "e223d64568524102ab1a3fc2bbd935b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804ab5a6027147b58c3017dba58648ff" is_pre_election: true
I20260925 02:56:45.808389 804 raft_consensus.cc:2454] T 3065d8d260a44d7989867fb60d464d40 P 804ab5a6027147b58c3017dba58648ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e223d64568524102ab1a3fc2bbd935b6 in term 0.
I20260925 02:56:45.808384 937 raft_consensus.cc:2454] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e223d64568524102ab1a3fc2bbd935b6 in term 0.
I20260925 02:56:45.808830 1003 leader_election.cc:302] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4431abb4fc45461c88eceb626631b19d, e223d64568524102ab1a3fc2bbd935b6; no voters:
I20260925 02:56:45.809044 1139 raft_consensus.cc:2781] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:56:45.809121 1139 raft_consensus.cc:482] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:56:45.809150 1139 raft_consensus.cc:3037] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:45.810137 1139 raft_consensus.cc:504] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4431abb4fc45461c88eceb626631b19d" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 37215 } } peers { permanent_uuid: "804ab5a6027147b58c3017dba58648ff" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41637 } } peers { permanent_uuid: "e223d64568524102ab1a3fc2bbd935b6" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45367 } }
I20260925 02:56:45.810474 1139 leader_election.cc:288] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [CANDIDATE]: Term 1 election: Requested vote from peers 4431abb4fc45461c88eceb626631b19d (127.12.222.194:37215), 804ab5a6027147b58c3017dba58648ff (127.12.222.193:41637)
I20260925 02:56:45.810822 937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3065d8d260a44d7989867fb60d464d40" candidate_uuid: "e223d64568524102ab1a3fc2bbd935b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4431abb4fc45461c88eceb626631b19d"
I20260925 02:56:45.810860 804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3065d8d260a44d7989867fb60d464d40" candidate_uuid: "e223d64568524102ab1a3fc2bbd935b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804ab5a6027147b58c3017dba58648ff"
I20260925 02:56:45.810958 804 raft_consensus.cc:3037] T 3065d8d260a44d7989867fb60d464d40 P 804ab5a6027147b58c3017dba58648ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:45.810958 937 raft_consensus.cc:3037] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:45.812131 937 raft_consensus.cc:2454] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e223d64568524102ab1a3fc2bbd935b6 in term 1.
I20260925 02:56:45.812131 804 raft_consensus.cc:2454] T 3065d8d260a44d7989867fb60d464d40 P 804ab5a6027147b58c3017dba58648ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e223d64568524102ab1a3fc2bbd935b6 in term 1.
I20260925 02:56:45.812412 1003 leader_election.cc:302] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 804ab5a6027147b58c3017dba58648ff, e223d64568524102ab1a3fc2bbd935b6; no voters:
I20260925 02:56:45.812580 1139 raft_consensus.cc:2781] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:45.812826 1139 raft_consensus.cc:686] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [term 1 LEADER]: Becoming Leader. State: Replica: e223d64568524102ab1a3fc2bbd935b6, State: Running, Role: LEADER
I20260925 02:56:45.813127 1139 consensus_queue.cc:237] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4431abb4fc45461c88eceb626631b19d" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 37215 } } peers { permanent_uuid: "804ab5a6027147b58c3017dba58648ff" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41637 } } peers { permanent_uuid: "e223d64568524102ab1a3fc2bbd935b6" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45367 } }
I20260925 02:56:45.815981 661 catalog_manager.cc:5495] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e223d64568524102ab1a3fc2bbd935b6 (127.12.222.195). New cstate: current_term: 1 leader_uuid: "e223d64568524102ab1a3fc2bbd935b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4431abb4fc45461c88eceb626631b19d" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 37215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "804ab5a6027147b58c3017dba58648ff" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e223d64568524102ab1a3fc2bbd935b6" member_type: VOTER last_known_addr { host: "127.12.222.195" port: 45367 } health_report { overall_health: HEALTHY } } }
I20260925 02:56:45.870784 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.867294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53182 (local address 127.12.222.195:45367)
0925 02:56:45.867429 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:45.867433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:45.867447 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:56:45.867597 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:45.867600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:45.867662 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:45.867768 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:56:45.868016 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.868541 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.869386 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:45.869613 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:45.869714 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:45.870009 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:56:45.870371 (+ 362us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:56:45.870498 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:45.870553 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 02:56:45.870601 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:56:45.876067 937 raft_consensus.cc:1261] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d [term 1 FOLLOWER]: Refusing update from remote peer e223d64568524102ab1a3fc2bbd935b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:45.876067 804 raft_consensus.cc:1261] T 3065d8d260a44d7989867fb60d464d40 P 804ab5a6027147b58c3017dba58648ff [term 1 FOLLOWER]: Refusing update from remote peer e223d64568524102ab1a3fc2bbd935b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:56:45.876783 1139 consensus_queue.cc:1035] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "804ab5a6027147b58c3017dba58648ff" member_type: VOTER last_known_addr { host: "127.12.222.193" port: 41637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:45.876994 1143 consensus_queue.cc:1035] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4431abb4fc45461c88eceb626631b19d" member_type: VOTER last_known_addr { host: "127.12.222.194" port: 37215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:56:45.883047 1146 mvcc.cc:205] Tried to move back new op lower bound from 7333089304061902848 to 7333089303812816896. Current Snapshot: MvccSnapshot[applied={T|T < 7333089304061902848}]
I20260925 02:56:45.883339 1148 mvcc.cc:205] Tried to move back new op lower bound from 7333089304061902848 to 7333089303812816896. Current Snapshot: MvccSnapshot[applied={T|T < 7333089304061902848}]
I20260925 02:56:45.891283 659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:45.891397 659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:56:45.893482 659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9757684f8e14d64949daf6a5f100a01: Sending DeleteTablet for 3 replicas of tablet 3065d8d260a44d7989867fb60d464d40
I20260925 02:56:45.894255 1049 tablet_service.cc:1514] Processing DeleteTablet for tablet 3065d8d260a44d7989867fb60d464d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:45 UTC) from {username='slave'} at 127.0.0.1:53172
I20260925 02:56:45.894353 784 tablet_service.cc:1514] Processing DeleteTablet for tablet 3065d8d260a44d7989867fb60d464d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:45 UTC) from {username='slave'} at 127.0.0.1:36720
I20260925 02:56:45.894330 917 tablet_service.cc:1514] Processing DeleteTablet for tablet 3065d8d260a44d7989867fb60d464d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:56:45 UTC) from {username='slave'} at 127.0.0.1:39588
I20260925 02:56:45.894711 1155 tablet_replica.cc:330] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6: stopping tablet replica
I20260925 02:56:45.894711 1156 tablet_replica.cc:330] T 3065d8d260a44d7989867fb60d464d40 P 804ab5a6027147b58c3017dba58648ff: stopping tablet replica
I20260925 02:56:45.894990 1156 raft_consensus.cc:2229] T 3065d8d260a44d7989867fb60d464d40 P 804ab5a6027147b58c3017dba58648ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:45.894990 1155 raft_consensus.cc:2229] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:56:45.895273 1155 raft_consensus.cc:2258] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:45.895268 1156 raft_consensus.cc:2258] T 3065d8d260a44d7989867fb60d464d40 P 804ab5a6027147b58c3017dba58648ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:45.895490 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 722
I20260925 02:56:45.897882 1155 ts_tablet_manager.cc:1905] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:45.900278 1155 ts_tablet_manager.cc:1918] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:56:45.900388 1155 log.cc:1196] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-2/wal/wals/3065d8d260a44d7989867fb60d464d40
I20260925 02:56:45.900427 1157 tablet_replica.cc:330] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d: stopping tablet replica
I20260925 02:56:45.900696 1155 ts_tablet_manager.cc:1939] T 3065d8d260a44d7989867fb60d464d40 P e223d64568524102ab1a3fc2bbd935b6: Deleting consensus metadata
I20260925 02:56:45.900707 1157 raft_consensus.cc:2229] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:56:45.900981 1157 raft_consensus.cc:2258] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:56:45.902148 648 catalog_manager.cc:4841] TS e223d64568524102ab1a3fc2bbd935b6 (127.12.222.195:45367): tablet 3065d8d260a44d7989867fb60d464d40 (table test-table [id=b67e254442354c80a600917de8a3a8f4]) successfully deleted
I20260925 02:56:45.902257 1157 ts_tablet_manager.cc:1905] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:56:45.904265 1157 ts_tablet_manager.cc:1918] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:56:45.904366 1157 log.cc:1196] T 3065d8d260a44d7989867fb60d464d40 P 4431abb4fc45461c88eceb626631b19d: Deleting WAL directory at /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304762911399-13179-0/minicluster-data/ts-1/wal/wals/3065d8d260a44d7989867fb60d464d40
W20260925 02:56:45.904377 646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.222.193:41637 (error 108)
I20260925 02:56:45.904664 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 853
I20260925 02:56:45.906049 1130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.905740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.193:41637 (local address 127.0.0.1:36724)
0925 02:56:45.905909 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:45.905973 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.193:41637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260925 02:56:45.906190 646 catalog_manager.cc:4584] TS 804ab5a6027147b58c3017dba58648ff (127.12.222.193:41637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3065d8d260a44d7989867fb60d464d40: Network error: Client connection negotiation failed: client connection to 127.12.222.193:41637: connect: Connection refused (error 111)
I20260925 02:56:45.910039 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 987
I20260925 02:56:45.911563 1130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:45.910247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.194:37215 (local address 127.0.0.1:39596)
0925 02:56:45.911466 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0925 02:56:45.911509 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.194:37215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1144}
I20260925 02:56:45.915455 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 630
2026-09-25T02:56:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1097 ms)
[----------] 6 tests from EncryptionPolicyTest (10042 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:45 dist-test-slave-252r krb5kdc[1161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:45 dist-test-slave-252r krb5kdc[1161](info): set up 2 sockets
Sep 25 02:56:45 dist-test-slave-252r krb5kdc[1161](info): commencing operation
krb5kdc: starting...
W20260925 02:56:46.928457 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:56:47 dist-test-slave-252r krb5kdc[1161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:56:47.038499 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33905
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:47.138782 1175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:47.138983 1175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:47.139019 1175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:47.142704 1175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:47.142745 1175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:47.142763 1175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:47.142777 1175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:47.142794 1175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:47.147404 1175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33905
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:47.148275 1175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:47.149114 1175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:47.156582 1181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:47.156594 1183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:47.156618 1180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:47.156627 1175 server_base.cc:1034] running on GCE node
I20260925 02:56:47.157231 1175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:47.158128 1175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:47.159307 1175 hybrid_clock.cc:648] HybridClock initialized: now 1790305007159297 us; error 43 us; skew 500 ppm
Sep 25 02:56:47 dist-test-slave-252r krb5kdc[1161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:47.161777 1175 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:56:47.162472 1175 webserver.cc:460] Webserver started at http://127.0.0.1:35803/ using document root <none> and password file <none>
I20260925 02:56:47.162884 1175 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:47.162935 1175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:47.163069 1175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:47.164808 1175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "8cc621e5b21e49e48711fcd66057613b"
format_stamp: "Formatted at 2026-09-25 02:56:47 on dist-test-slave-252r"
server_key: "db88e0891e4f3042327ff223c65f6ccd"
server_key_iv: "15098ad229618793e46d9d50ac36b954"
server_key_version: "encryptionkey@0"
I20260925 02:56:47.165299 1175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "8cc621e5b21e49e48711fcd66057613b"
format_stamp: "Formatted at 2026-09-25 02:56:47 on dist-test-slave-252r"
server_key: "db88e0891e4f3042327ff223c65f6ccd"
server_key_iv: "15098ad229618793e46d9d50ac36b954"
server_key_version: "encryptionkey@0"
I20260925 02:56:47.168504 1175 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:56:47.170557 1190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:47.171619 1175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:47.171706 1175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "8cc621e5b21e49e48711fcd66057613b"
format_stamp: "Formatted at 2026-09-25 02:56:47 on dist-test-slave-252r"
server_key: "db88e0891e4f3042327ff223c65f6ccd"
server_key_iv: "15098ad229618793e46d9d50ac36b954"
server_key_version: "encryptionkey@0"
I20260925 02:56:47.171777 1175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:47.192313 1175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:47.194810 1175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:47.194943 1175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:47.201517 1175 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33905
I20260925 02:56:47.201540 1242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33905 every 8 connection(s)
I20260925 02:56:47.202415 1175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:47.203358 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 1175
I20260925 02:56:47.203488 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:56:47.203774 13179 external_mini_cluster.cc:1428] Setting key f1a2caa334651a681855d809ec7546e7
I20260925 02:56:47.205612 1243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:47.210482 1243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b: Bootstrap starting.
Sep 25 02:56:47 dist-test-slave-252r krb5kdc[1161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:56:47.212491 1243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:47.213138 1243 log.cc:824] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:47.215169 1243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b: No bootstrap required, opened a new log
I20260925 02:56:47.217126 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:47.205118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40575 (local address 127.0.0.1:33905)
0925 02:56:47.205553 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:47.205566 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:47.205598 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:56:47.206250 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:47.206265 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:47.206557 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:47.206855 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:47.207608 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:47.208559 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:47.209354 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:47.209798 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:47.209967 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:47.212314 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:47.212320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:47.212337 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:47.212361 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:47.214168 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:47.214663 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:47.214668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:47.214674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:47.214808 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:47.215130 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:47.215133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:47.215135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:47.215592 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:47.215796 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:47.216036 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 02:56:47.216258 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":159,"threads_started":1}
I20260925 02:56:47.219252 1243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc621e5b21e49e48711fcd66057613b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33905 } }
I20260925 02:56:47.219491 1243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:47.219564 1243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc621e5b21e49e48711fcd66057613b, State: Initialized, Role: FOLLOWER
I20260925 02:56:47.219937 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:56:47.219972 1243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc621e5b21e49e48711fcd66057613b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33905 } }
I20260925 02:56:47.220139 1243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:47.220198 1243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:47.220320 1243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:47.221199 1243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc621e5b21e49e48711fcd66057613b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33905 } }
I20260925 02:56:47.221519 1243 leader_election.cc:302] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cc621e5b21e49e48711fcd66057613b; no voters:
I20260925 02:56:47.221854 1243 leader_election.cc:288] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:47.222066 1252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:47.222325 1252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b [term 1 LEADER]: Becoming Leader. State: Replica: 8cc621e5b21e49e48711fcd66057613b, State: Running, Role: LEADER
I20260925 02:56:47.222633 1252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc621e5b21e49e48711fcd66057613b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33905 } }
I20260925 02:56:47.223014 1243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:47.224198 1254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cc621e5b21e49e48711fcd66057613b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc621e5b21e49e48711fcd66057613b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33905 } } }
I20260925 02:56:47.224354 1254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:47.224592 1254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cc621e5b21e49e48711fcd66057613b. Latest consensus state: current_term: 1 leader_uuid: "8cc621e5b21e49e48711fcd66057613b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc621e5b21e49e48711fcd66057613b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33905 } } }
I20260925 02:56:47.224695 1254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b [sys.catalog]: This master's current role is: LEADER
W20260925 02:56:47.228000 1268 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:56:47.228075 1268 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:56:47.228147 1263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:47.228991 1263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:47.230391 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:47.221872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52383 (local address 127.0.0.1:33905)
0925 02:56:47.221961 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:47.221965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:47.222173 (+ 208us) server_negotiation.cc:407] Connection header received
0925 02:56:47.222336 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:47.222341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:47.222380 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:47.222435 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:47.223206 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:47.223878 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:47.225534 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:47.225852 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:47.225966 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:47.227251 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:47.227254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:47.227257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:47.227279 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:47.228664 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:47.229171 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:47.229173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:47.229174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:47.229240 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:47.229575 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:47.229578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:47.229580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:47.229736 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:47.229888 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:47.230145 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 02:56:47.230229 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 02:56:47.234398 1263 catalog_manager.cc:1347] Generated new cluster ID: 0931a92e2b3a478fb93cc9121c766008
I20260925 02:56:47.234467 1263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:47.252556 1263 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:47.253306 1263 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:47.262051 1263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cc621e5b21e49e48711fcd66057613b: Generated new TSK 0
I20260925 02:56:47.262738 1263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:47.272330 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 1175
W20260925 02:56:47.278028 1159 connection.cc:535] client connection to 127.0.0.1:33905 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33905): Connection reset by peer (error 104)
2026-09-25T02:56:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:47 dist-test-slave-252r krb5kdc[1274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:47 dist-test-slave-252r krb5kdc[1274](info): set up 2 sockets
Sep 25 02:56:47 dist-test-slave-252r krb5kdc[1274](info): commencing operation
krb5kdc: starting...
W20260925 02:56:48.251972 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:56:48 dist-test-slave-252r krb5kdc[1274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:56:48.359225 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:48.458448 1287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:48.458680 1287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:48.458752 1287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:48.462438 1287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:48.462509 1287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:48.462541 1287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:48.462608 1287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:48.462652 1287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:48.467185 1287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33769
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:48.468001 1287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:48.468927 1287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:48.476404 1292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:48.476414 1293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:48.476492 1287 server_base.cc:1034] running on GCE node
W20260925 02:56:48.476449 1295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:48.477093 1287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:48.477983 1287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:48.479126 1287 hybrid_clock.cc:648] HybridClock initialized: now 1790305008479105 us; error 34 us; skew 500 ppm
Sep 25 02:56:48 dist-test-slave-252r krb5kdc[1274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:48.481756 1287 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:56:48.482493 1287 webserver.cc:460] Webserver started at http://127.0.0.1:43823/ using document root <none> and password file <none>
I20260925 02:56:48.482940 1287 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:48.482995 1287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:48.483135 1287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:48.484972 1287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "a5474e6e13ef4233a58e880450cf8014"
format_stamp: "Formatted at 2026-09-25 02:56:48 on dist-test-slave-252r"
server_key: "2bdfb62c0c5d519665fe1c3317bca2b1"
server_key_iv: "bbfe0ca8be4d142619c1d70578830159"
server_key_version: "encryptionkey@0"
I20260925 02:56:48.485436 1287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "a5474e6e13ef4233a58e880450cf8014"
format_stamp: "Formatted at 2026-09-25 02:56:48 on dist-test-slave-252r"
server_key: "2bdfb62c0c5d519665fe1c3317bca2b1"
server_key_iv: "bbfe0ca8be4d142619c1d70578830159"
server_key_version: "encryptionkey@0"
I20260925 02:56:48.488706 1287 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:56:48.490720 1302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:48.491801 1287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:48.491936 1287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "a5474e6e13ef4233a58e880450cf8014"
format_stamp: "Formatted at 2026-09-25 02:56:48 on dist-test-slave-252r"
server_key: "2bdfb62c0c5d519665fe1c3317bca2b1"
server_key_iv: "bbfe0ca8be4d142619c1d70578830159"
server_key_version: "encryptionkey@0"
I20260925 02:56:48.492055 1287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:48.506533 1287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:48.509420 1287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:48.509593 1287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:48.515714 1287 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33769
I20260925 02:56:48.515769 1355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33769 every 8 connection(s)
I20260925 02:56:48.516587 1287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:48.518934 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:48.523762 1356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014: Bootstrap starting.
I20260925 02:56:48.524130 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 1287
I20260925 02:56:48.524297 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:56:48.524559 13179 external_mini_cluster.cc:1428] Setting key 01f59c0626777bbc4fd436193d96889b
I20260925 02:56:48.526196 1356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:48.527004 1356 log.cc:824] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:48.529433 1356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014: No bootstrap required, opened a new log
I20260925 02:56:48.533350 1356 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5474e6e13ef4233a58e880450cf8014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33769 } }
I20260925 02:56:48.533579 1356 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:48.533633 1356 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5474e6e13ef4233a58e880450cf8014, State: Initialized, Role: FOLLOWER
I20260925 02:56:48.533795 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:48.525783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56905 (local address 127.0.0.1:33769)
0925 02:56:48.526188 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:48.526208 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:48.526251 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:56:48.527047 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:48.527071 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:48.527477 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:48.527805 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:48.528218 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:48.529557 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:48.530411 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:48.531098 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:48.531258 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:48.532025 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:48.532033 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:48.532049 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:48.532515 (+ 466us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:48.532573 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:48.532599 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:48.532719 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:48.532866 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 02:56:48.532981 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":101,"threads_started":1}
I20260925 02:56:48.534060 1356 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5474e6e13ef4233a58e880450cf8014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33769 } }
I20260925 02:56:48.534205 1356 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:48.534266 1356 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:48.534355 1356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:48.535215 1356 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5474e6e13ef4233a58e880450cf8014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33769 } }
I20260925 02:56:48.535597 1356 leader_election.cc:302] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5474e6e13ef4233a58e880450cf8014; no voters:
I20260925 02:56:48.535907 1356 leader_election.cc:288] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:48.536033 1361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:48.536288 1361 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014 [term 1 LEADER]: Becoming Leader. State: Replica: a5474e6e13ef4233a58e880450cf8014, State: Running, Role: LEADER
I20260925 02:56:48.536659 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:56:48.536636 1361 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5474e6e13ef4233a58e880450cf8014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33769 } }
I20260925 02:56:48.537186 1356 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:48.537755 1361 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5474e6e13ef4233a58e880450cf8014. Latest consensus state: current_term: 1 leader_uuid: "a5474e6e13ef4233a58e880450cf8014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5474e6e13ef4233a58e880450cf8014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33769 } } }
I20260925 02:56:48.537878 1361 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:48.538237 1362 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5474e6e13ef4233a58e880450cf8014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5474e6e13ef4233a58e880450cf8014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33769 } } }
I20260925 02:56:48.538334 1362 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014 [sys.catalog]: This master's current role is: LEADER
W20260925 02:56:48.541728 1381 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:56:48.541815 1381 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:56:48.542032 1369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:48.542925 1369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:48.543641 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:48.538975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58469 (local address 127.0.0.1:33769)
0925 02:56:48.539070 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:48.539074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:48.539469 (+ 395us) server_negotiation.cc:407] Connection header received
0925 02:56:48.539629 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:48.539632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:48.539669 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:48.539730 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:48.540142 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:48.540911 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:48.542004 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:48.542296 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:48.542434 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:48.543230 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:48.543233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:48.543235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:48.543246 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:48.543270 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:48.543285 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:48.543372 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:48.543487 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 02:56:48.543515 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:56:48.547801 1369 catalog_manager.cc:1347] Generated new cluster ID: 2812b086be8347b889afc7e53f219da7
I20260925 02:56:48.547863 1369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:48.565515 1369 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:48.566442 1369 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:48.573786 1369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5474e6e13ef4233a58e880450cf8014: Generated new TSK 0
I20260925 02:56:48.574406 1369 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:48.583336 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 1287
W20260925 02:56:48.588406 1272 connection.cc:535] client connection to 127.0.0.1:33769 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33769): Connection reset by peer (error 104)
2026-09-25T02:56:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:48 dist-test-slave-252r krb5kdc[1387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:48 dist-test-slave-252r krb5kdc[1387](info): set up 2 sockets
Sep 25 02:56:48 dist-test-slave-252r krb5kdc[1387](info): commencing operation
krb5kdc: starting...
W20260925 02:56:49.564097 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:56:49 dist-test-slave-252r krb5kdc[1387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:56:49.670511 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:49.769797 1400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:49.769997 1400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:49.770031 1400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:49.773530 1400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:49.773579 1400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:49.773597 1400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:49.773627 1400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:49.773643 1400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:49.778147 1400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46571
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:49.778913 1400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:49.779734 1400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:49.787151 1408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:49.787143 1406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:49.787294 1400 server_base.cc:1034] running on GCE node
W20260925 02:56:49.787463 1405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:49.787968 1400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:49.788995 1400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:49.790151 1400 hybrid_clock.cc:648] HybridClock initialized: now 1790305009790134 us; error 29 us; skew 500 ppm
Sep 25 02:56:49 dist-test-slave-252r krb5kdc[1387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:49.792582 1400 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:56:49.793368 1400 webserver.cc:460] Webserver started at http://127.0.0.1:36787/ using document root <none> and password file <none>
I20260925 02:56:49.793871 1400 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:49.793962 1400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:49.794170 1400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:49.796053 1400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "bfd08714d39c4509875bc8f68067e908"
format_stamp: "Formatted at 2026-09-25 02:56:49 on dist-test-slave-252r"
server_key: "22c9c14feb2034e364fd78720a1ba4dd"
server_key_iv: "b0b1a517d17792c93d38b72c7340323e"
server_key_version: "encryptionkey@0"
I20260925 02:56:49.796626 1400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "bfd08714d39c4509875bc8f68067e908"
format_stamp: "Formatted at 2026-09-25 02:56:49 on dist-test-slave-252r"
server_key: "22c9c14feb2034e364fd78720a1ba4dd"
server_key_iv: "b0b1a517d17792c93d38b72c7340323e"
server_key_version: "encryptionkey@0"
I20260925 02:56:49.799990 1400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:56:49.802181 1415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:49.803220 1400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 02:56:49.803372 1400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "bfd08714d39c4509875bc8f68067e908"
format_stamp: "Formatted at 2026-09-25 02:56:49 on dist-test-slave-252r"
server_key: "22c9c14feb2034e364fd78720a1ba4dd"
server_key_iv: "b0b1a517d17792c93d38b72c7340323e"
server_key_version: "encryptionkey@0"
I20260925 02:56:49.803474 1400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:49.807585 1400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:49.810089 1400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:49.810258 1400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:49.816629 1400 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46571
I20260925 02:56:49.816638 1468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46571 every 8 connection(s)
I20260925 02:56:49.817374 1400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:49.819747 1469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:49.824523 1469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908: Bootstrap starting.
I20260925 02:56:49.826097 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 1400
I20260925 02:56:49.826232 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:56:49.826503 13179 external_mini_cluster.cc:1428] Setting key 08e3eb65c10a1ec94ed7525820318ef7
I20260925 02:56:49.826771 1469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:49.827576 1469 log.cc:824] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:49.829465 1469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908: No bootstrap required, opened a new log
I20260925 02:56:49.831010 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:49.827447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40529 (local address 127.0.0.1:46571)
0925 02:56:49.827801 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:49.827814 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:49.827843 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:56:49.828607 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:49.828625 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:49.828865 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:49.829134 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:49.829933 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:49.829938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:49.829948 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:49.829957 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:49.830004 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:49.830020 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:49.830104 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:49.830248 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 02:56:49.830394 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":118,"threads_started":1}
I20260925 02:56:49.833227 1469 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd08714d39c4509875bc8f68067e908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46571 } }
I20260925 02:56:49.833474 1469 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:49.833575 1469 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfd08714d39c4509875bc8f68067e908, State: Initialized, Role: FOLLOWER
I20260925 02:56:49.833683 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:56:49.833988 1469 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd08714d39c4509875bc8f68067e908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46571 } }
I20260925 02:56:49.834110 1469 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:49.834193 1469 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:49.834326 1469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:49.835167 1469 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd08714d39c4509875bc8f68067e908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46571 } }
I20260925 02:56:49.835482 1469 leader_election.cc:302] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfd08714d39c4509875bc8f68067e908; no voters:
I20260925 02:56:49.835807 1469 leader_election.cc:288] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:49.835855 1479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:49.836112 1479 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908 [term 1 LEADER]: Becoming Leader. State: Replica: bfd08714d39c4509875bc8f68067e908, State: Running, Role: LEADER
I20260925 02:56:49.836485 1479 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd08714d39c4509875bc8f68067e908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46571 } }
I20260925 02:56:49.837044 1469 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:49.837636 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:49.835815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55079 (local address 127.0.0.1:46571)
0925 02:56:49.835953 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:49.835957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:49.836342 (+ 385us) server_negotiation.cc:407] Connection header received
0925 02:56:49.836388 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:49.836390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:49.836418 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:49.836480 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:49.837221 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:49.837224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:49.837226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:49.837236 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:49.837258 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:49.837268 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:49.837340 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:49.837499 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 02:56:49.837531 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:56:49.838261 1482 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfd08714d39c4509875bc8f68067e908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd08714d39c4509875bc8f68067e908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46571 } } }
I20260925 02:56:49.838321 1481 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfd08714d39c4509875bc8f68067e908. Latest consensus state: current_term: 1 leader_uuid: "bfd08714d39c4509875bc8f68067e908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd08714d39c4509875bc8f68067e908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46571 } } }
I20260925 02:56:49.838378 1482 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:49.838378 1481 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:49.838771 1489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:49.841784 1489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:49.846784 1489 catalog_manager.cc:1347] Generated new cluster ID: 0b55a654ec104d3a8bdb4296b27045aa
I20260925 02:56:49.846845 1489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:49.858199 1489 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:49.858474 1489 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:49.866618 1489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfd08714d39c4509875bc8f68067e908: Generated new TSK 0
I20260925 02:56:49.867269 1489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:49.874469 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 1400
2026-09-25T02:56:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:49 dist-test-slave-252r krb5kdc[1501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:49 dist-test-slave-252r krb5kdc[1501](info): set up 2 sockets
Sep 25 02:56:49 dist-test-slave-252r krb5kdc[1501](info): commencing operation
krb5kdc: starting...
W20260925 02:56:50.848484 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:56:50 dist-test-slave-252r krb5kdc[1501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:56:50.954859 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39661
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:51.053983 1514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:51.054231 1514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:51.054311 1514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:51.058099 1514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:51.058169 1514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:51.058204 1514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:51.058255 1514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:51.058290 1514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:51.062903 1514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:51.063721 1514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:51.064667 1514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:51.072052 1520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:51.072207 1514 server_base.cc:1034] running on GCE node
W20260925 02:56:51.072057 1519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:51.072063 1522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:51.072841 1514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:51.073665 1514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:51.074822 1514 hybrid_clock.cc:648] HybridClock initialized: now 1790305011074806 us; error 26 us; skew 500 ppm
Sep 25 02:56:51 dist-test-slave-252r krb5kdc[1501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:51.080550 1514 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:56:51.081362 1514 webserver.cc:460] Webserver started at http://127.0.0.1:43561/ using document root <none> and password file <none>
I20260925 02:56:51.081847 1514 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:51.081902 1514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:51.082140 1514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:51.083760 1514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "16fd44dd0a594740ab00a7006964f080"
format_stamp: "Formatted at 2026-09-25 02:56:51 on dist-test-slave-252r"
server_key: "4b01c43dabf9c4bc8c9c83ce6e827910"
server_key_iv: "d6111d51aed067581f793b763d1d105d"
server_key_version: "encryptionkey@0"
I20260925 02:56:51.084375 1514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "16fd44dd0a594740ab00a7006964f080"
format_stamp: "Formatted at 2026-09-25 02:56:51 on dist-test-slave-252r"
server_key: "4b01c43dabf9c4bc8c9c83ce6e827910"
server_key_iv: "d6111d51aed067581f793b763d1d105d"
server_key_version: "encryptionkey@0"
I20260925 02:56:51.087674 1514 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:56:51.089726 1529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:51.090749 1514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:56:51.090888 1514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "16fd44dd0a594740ab00a7006964f080"
format_stamp: "Formatted at 2026-09-25 02:56:51 on dist-test-slave-252r"
server_key: "4b01c43dabf9c4bc8c9c83ce6e827910"
server_key_iv: "d6111d51aed067581f793b763d1d105d"
server_key_version: "encryptionkey@0"
I20260925 02:56:51.091013 1514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:51.116852 1514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:51.119225 1514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:51.119395 1514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:51.126017 1514 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39661
I20260925 02:56:51.126041 1581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39661 every 8 connection(s)
I20260925 02:56:51.126817 1514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:51.129173 1582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:51.130134 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 1514
I20260925 02:56:51.130259 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:56:51.130565 13179 external_mini_cluster.cc:1428] Setting key 612bee1781d3ee96a6b6a9e444a8533a
I20260925 02:56:51.134414 1582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080: Bootstrap starting.
I20260925 02:56:51.136407 1582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:51.137034 1582 log.cc:824] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:56:51 dist-test-slave-252r krb5kdc[1501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:56:51.138924 1582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080: No bootstrap required, opened a new log
I20260925 02:56:51.141664 1582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fd44dd0a594740ab00a7006964f080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39661 } }
I20260925 02:56:51.141959 1582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:51.142026 1582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16fd44dd0a594740ab00a7006964f080, State: Initialized, Role: FOLLOWER
I20260925 02:56:51.142472 1582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fd44dd0a594740ab00a7006964f080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39661 } }
I20260925 02:56:51.142614 1582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:51.142674 1582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:51.142755 1582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:51.143685 1582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fd44dd0a594740ab00a7006964f080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39661 } }
I20260925 02:56:51.143987 1582 leader_election.cc:302] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16fd44dd0a594740ab00a7006964f080; no voters:
I20260925 02:56:51.144342 1582 leader_election.cc:288] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:51.144449 1587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:51.144565 1585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:51.131783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37665 (local address 127.0.0.1:39661)
0925 02:56:51.132185 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:51.132198 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:51.132262 (+ 64us) server_negotiation.cc:407] Connection header received
0925 02:56:51.132831 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:51.132846 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:51.133131 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:51.133417 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:51.134161 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:51.135101 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:51.135953 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:51.136418 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:51.136587 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:51.139193 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:51.139198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:51.139208 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:51.139231 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:51.141274 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:51.141838 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:51.141845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:51.141852 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:51.142029 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:51.142377 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:51.142381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:51.142382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:51.142836 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:51.143011 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:51.143404 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 02:56:51.143671 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":119,"threads_started":1}
I20260925 02:56:51.144649 1587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080 [term 1 LEADER]: Becoming Leader. State: Replica: 16fd44dd0a594740ab00a7006964f080, State: Running, Role: LEADER
I20260925 02:56:51.144909 1587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fd44dd0a594740ab00a7006964f080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39661 } }
I20260925 02:56:51.145556 1582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:51.146051 1587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16fd44dd0a594740ab00a7006964f080. Latest consensus state: current_term: 1 leader_uuid: "16fd44dd0a594740ab00a7006964f080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fd44dd0a594740ab00a7006964f080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39661 } } }
I20260925 02:56:51.146767 1587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:51.146651 1588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16fd44dd0a594740ab00a7006964f080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fd44dd0a594740ab00a7006964f080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39661 } } }
I20260925 02:56:51.147042 1588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:51.147104 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 02:56:51.150336 1602 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:56:51.150419 1602 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:56:51.150525 1596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:51.151207 1596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:51.156450 1596 catalog_manager.cc:1347] Generated new cluster ID: 3630837b68604852abe880e00e5baf23
I20260925 02:56:51.156503 1596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:51.157770 1585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:51.149123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39429 (local address 127.0.0.1:39661)
0925 02:56:51.149239 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:51.149243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:51.149611 (+ 368us) server_negotiation.cc:407] Connection header received
0925 02:56:51.149696 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:51.149698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:51.149753 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:51.149827 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:51.150532 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:51.151272 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:51.152506 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:51.152841 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:51.152940 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:51.154402 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:51.154406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:51.154408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:51.154430 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:51.156114 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:51.156650 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:51.156653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:51.156655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:51.156733 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:51.156993 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:51.156996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:51.156997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:51.157149 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:51.157268 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:51.157487 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 02:56:51.157585 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:56:51.184221 1596 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:51.185108 1596 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:51.195664 1596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16fd44dd0a594740ab00a7006964f080: Generated new TSK 0
I20260925 02:56:51.196324 1596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:51.200325 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 1514
2026-09-25T02:56:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:51 dist-test-slave-252r krb5kdc[1613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:51 dist-test-slave-252r krb5kdc[1613](info): set up 2 sockets
Sep 25 02:56:51 dist-test-slave-252r krb5kdc[1613](info): commencing operation
krb5kdc: starting...
W20260925 02:56:52.160483 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:56:52 dist-test-slave-252r krb5kdc[1613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:56:52.265031 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:52.363261 1626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:52.363468 1626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:52.363530 1626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:52.367338 1626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:52.367383 1626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:52.367403 1626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:52.367456 1626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:52.367488 1626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:52.372176 1626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:52.372998 1626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:52.373857 1626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:52.381397 1634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:52.381453 1626 server_base.cc:1034] running on GCE node
W20260925 02:56:52.381445 1631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:52.381407 1632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:52.382143 1626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:52.383100 1626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:52.384301 1626 hybrid_clock.cc:648] HybridClock initialized: now 1790305012384274 us; error 38 us; skew 500 ppm
Sep 25 02:56:52 dist-test-slave-252r krb5kdc[1613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:52.386847 1626 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:56:52.387532 1626 webserver.cc:460] Webserver started at http://127.0.0.1:37239/ using document root <none> and password file <none>
I20260925 02:56:52.387969 1626 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:52.388029 1626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:52.388166 1626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:52.389909 1626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "7af3f15f5e5e495481a33144a00e746c"
format_stamp: "Formatted at 2026-09-25 02:56:52 on dist-test-slave-252r"
server_key: "b103496321115c7b81af3e554ed8290c"
server_key_iv: "9d6412fabaeb214e5238421e6bc01c3b"
server_key_version: "encryptionkey@0"
I20260925 02:56:52.390360 1626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "7af3f15f5e5e495481a33144a00e746c"
format_stamp: "Formatted at 2026-09-25 02:56:52 on dist-test-slave-252r"
server_key: "b103496321115c7b81af3e554ed8290c"
server_key_iv: "9d6412fabaeb214e5238421e6bc01c3b"
server_key_version: "encryptionkey@0"
I20260925 02:56:52.393453 1626 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:56:52.395430 1641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:52.396551 1626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:56:52.396668 1626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "7af3f15f5e5e495481a33144a00e746c"
format_stamp: "Formatted at 2026-09-25 02:56:52 on dist-test-slave-252r"
server_key: "b103496321115c7b81af3e554ed8290c"
server_key_iv: "9d6412fabaeb214e5238421e6bc01c3b"
server_key_version: "encryptionkey@0"
I20260925 02:56:52.396759 1626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:52.410948 1626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:52.413720 1626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:52.413863 1626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:52.419952 1626 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32781
I20260925 02:56:52.419992 1695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32781 every 8 connection(s)
I20260925 02:56:52.420778 1626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:52.423141 1696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:52.428041 1696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c: Bootstrap starting.
I20260925 02:56:52.430200 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 1626
I20260925 02:56:52.430323 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:56:52.430337 1696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:52.430577 13179 external_mini_cluster.cc:1428] Setting key 9b2963490b3b7651ab85147f64f20326
I20260925 02:56:52.431047 1696 log.cc:824] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:52.432929 1696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c: No bootstrap required, opened a new log
I20260925 02:56:52.437124 1696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af3f15f5e5e495481a33144a00e746c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32781 } }
I20260925 02:56:52.437358 1696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:52.437415 1696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af3f15f5e5e495481a33144a00e746c, State: Initialized, Role: FOLLOWER
I20260925 02:56:52.437907 1696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af3f15f5e5e495481a33144a00e746c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32781 } }
I20260925 02:56:52.438071 1696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:52.438135 1696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:52.438254 1696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:52.439128 1696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af3f15f5e5e495481a33144a00e746c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32781 } }
I20260925 02:56:52.439399 1700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:52.431544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52395 (local address 127.0.0.1:32781)
0925 02:56:52.431932 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:52.431947 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:52.431980 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:56:52.432618 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:52.432635 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:52.432927 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:52.433225 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:52.433593 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:52.434497 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:52.435860 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:52.436630 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:52.436785 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:52.437799 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:52.437805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:52.437820 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:52.438271 (+ 451us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:52.438338 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:52.438372 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:52.438458 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:52.438569 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 02:56:52.438749 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":115,"threads_started":1}
I20260925 02:56:52.439432 1696 leader_election.cc:302] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7af3f15f5e5e495481a33144a00e746c; no voters:
I20260925 02:56:52.439726 1696 leader_election.cc:288] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:52.439791 1701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:52.439968 1701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c [term 1 LEADER]: Becoming Leader. State: Replica: 7af3f15f5e5e495481a33144a00e746c, State: Running, Role: LEADER
I20260925 02:56:52.440294 1701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af3f15f5e5e495481a33144a00e746c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32781 } }
I20260925 02:56:52.440879 1696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:52.441448 1701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7af3f15f5e5e495481a33144a00e746c. Latest consensus state: current_term: 1 leader_uuid: "7af3f15f5e5e495481a33144a00e746c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af3f15f5e5e495481a33144a00e746c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32781 } } }
I20260925 02:56:52.441629 1701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:52.441835 1702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7af3f15f5e5e495481a33144a00e746c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af3f15f5e5e495481a33144a00e746c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32781 } } }
I20260925 02:56:52.441999 1702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:52.442063 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:56:52.442045 1709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:52.445135 1709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:52.449276 1700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:52.445069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58403 (local address 127.0.0.1:32781)
0925 02:56:52.445172 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:52.445176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:52.445670 (+ 494us) server_negotiation.cc:407] Connection header received
0925 02:56:52.445861 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:52.445864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:52.445913 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:52.445990 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:52.446315 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:52.447016 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:52.447770 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:52.447986 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:52.448084 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:52.448862 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:52.448868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:52.448871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:52.448882 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:52.448905 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:52.448917 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:52.448963 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:52.449134 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 02:56:52.449179 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:56:52.450652 1709 catalog_manager.cc:1347] Generated new cluster ID: 9fa2d355fd874564beda7d94e1a87d83
I20260925 02:56:52.450721 1709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:52.475946 1709 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:52.476809 1709 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:52.485512 1709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7af3f15f5e5e495481a33144a00e746c: Generated new TSK 0
I20260925 02:56:52.486174 1709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:52.490432 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 1626
2026-09-25T02:56:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:52 dist-test-slave-252r krb5kdc[1727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:52 dist-test-slave-252r krb5kdc[1727](info): set up 2 sockets
Sep 25 02:56:52 dist-test-slave-252r krb5kdc[1727](info): commencing operation
krb5kdc: starting...
W20260925 02:56:53.445128 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:56:53 dist-test-slave-252r krb5kdc[1727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:56:53.556210 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35659
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:53.656057 1740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:53.656275 1740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:53.656309 1740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:53.659672 1740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:53.659710 1740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:53.659729 1740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:53.659742 1740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:53.659756 1740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:53.664324 1740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:53.665263 1740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:53.666172 1740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:53.674676 1748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:53.674690 1746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:53.674690 1745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:53.675032 1740 server_base.cc:1034] running on GCE node
I20260925 02:56:53.675529 1740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:53.676512 1740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:53.677664 1740 hybrid_clock.cc:648] HybridClock initialized: now 1790305013677648 us; error 26 us; skew 500 ppm
Sep 25 02:56:53 dist-test-slave-252r krb5kdc[1727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:53.680225 1740 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:56:53.681038 1740 webserver.cc:460] Webserver started at http://127.0.0.1:38559/ using document root <none> and password file <none>
I20260925 02:56:53.681546 1740 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:53.681631 1740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:53.681845 1740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:53.683627 1740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "d745b9bf5b6e4cf79f73e09427cee7ef"
format_stamp: "Formatted at 2026-09-25 02:56:53 on dist-test-slave-252r"
server_key: "2eb50213e10bc609c86bb169fe558061"
server_key_iv: "b57912571e1dc74179bb279b18a1e1ad"
server_key_version: "encryptionkey@0"
I20260925 02:56:53.684191 1740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "d745b9bf5b6e4cf79f73e09427cee7ef"
format_stamp: "Formatted at 2026-09-25 02:56:53 on dist-test-slave-252r"
server_key: "2eb50213e10bc609c86bb169fe558061"
server_key_iv: "b57912571e1dc74179bb279b18a1e1ad"
server_key_version: "encryptionkey@0"
I20260925 02:56:53.687659 1740 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:56:53.689807 1755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:53.690851 1740 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 02:56:53.690982 1740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "d745b9bf5b6e4cf79f73e09427cee7ef"
format_stamp: "Formatted at 2026-09-25 02:56:53 on dist-test-slave-252r"
server_key: "2eb50213e10bc609c86bb169fe558061"
server_key_iv: "b57912571e1dc74179bb279b18a1e1ad"
server_key_version: "encryptionkey@0"
I20260925 02:56:53.691083 1740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:53.696942 1740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:53.700335 1740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:53.700522 1740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:53.707093 1740 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35659
I20260925 02:56:53.707154 1808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35659 every 8 connection(s)
I20260925 02:56:53.707926 1740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:53.710376 1809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:53.710762 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 1740
I20260925 02:56:53.710897 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:56:53.711149 13179 external_mini_cluster.cc:1428] Setting key 049f2839cb21ec23e2419b43d47faa4b
I20260925 02:56:53.716017 1812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:53.712326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33007 (local address 127.0.0.1:35659)
0925 02:56:53.712704 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:53.712721 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:53.712762 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:56:53.713454 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:53.713475 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:53.713791 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:53.714123 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:53.714736 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:53.714741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:53.714754 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:53.714764 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:53.714815 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:53.714839 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:53.714973 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:53.715040 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 02:56:53.715178 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
I20260925 02:56:53.717036 1809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef: Bootstrap starting.
I20260925 02:56:53.718706 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:56:53.719448 1809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:53.720340 1809 log.cc:824] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:53.722186 1809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef: No bootstrap required, opened a new log
I20260925 02:56:53.722450 1812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:53.720878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39505 (local address 127.0.0.1:35659)
0925 02:56:53.721015 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:53.721018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:53.721183 (+ 165us) server_negotiation.cc:407] Connection header received
0925 02:56:53.721307 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:53.721310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:53.721336 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:53.721423 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:53.722052 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:53.722055 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:53.722057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:53.722068 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:53.722091 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:53.722103 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:53.722192 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:53.722301 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 02:56:53.722337 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:56:53.725485 1809 raft_consensus.cc:348] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d745b9bf5b6e4cf79f73e09427cee7ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35659 } }
I20260925 02:56:53.725703 1809 raft_consensus.cc:374] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:53.725778 1809 raft_consensus.cc:729] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d745b9bf5b6e4cf79f73e09427cee7ef, State: Initialized, Role: FOLLOWER
I20260925 02:56:53.726142 1809 consensus_queue.cc:260] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d745b9bf5b6e4cf79f73e09427cee7ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35659 } }
I20260925 02:56:53.726279 1809 raft_consensus.cc:388] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:53.726361 1809 raft_consensus.cc:482] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:53.726451 1809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:53.727205 1809 raft_consensus.cc:504] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d745b9bf5b6e4cf79f73e09427cee7ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35659 } }
I20260925 02:56:53.727533 1809 leader_election.cc:302] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d745b9bf5b6e4cf79f73e09427cee7ef; no voters:
I20260925 02:56:53.727831 1809 leader_election.cc:288] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:53.727948 1820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:53.728159 1820 raft_consensus.cc:686] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef [term 1 LEADER]: Becoming Leader. State: Replica: d745b9bf5b6e4cf79f73e09427cee7ef, State: Running, Role: LEADER
I20260925 02:56:53.728451 1820 consensus_queue.cc:237] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d745b9bf5b6e4cf79f73e09427cee7ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35659 } }
I20260925 02:56:53.729025 1809 sys_catalog.cc:564] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:53.730052 1821 sys_catalog.cc:455] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d745b9bf5b6e4cf79f73e09427cee7ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d745b9bf5b6e4cf79f73e09427cee7ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35659 } } }
I20260925 02:56:53.730173 1821 sys_catalog.cc:458] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:53.730445 1822 sys_catalog.cc:455] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader d745b9bf5b6e4cf79f73e09427cee7ef. Latest consensus state: current_term: 1 leader_uuid: "d745b9bf5b6e4cf79f73e09427cee7ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d745b9bf5b6e4cf79f73e09427cee7ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35659 } } }
I20260925 02:56:53.730532 1822 sys_catalog.cc:458] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:53.730515 1829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:53.733042 1829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:53.737783 1829 catalog_manager.cc:1347] Generated new cluster ID: 00f961d06316491ca284d6f861a9d74f
I20260925 02:56:53.737851 1829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:53.757956 1829 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:53.758275 1829 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:53.775014 1829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d745b9bf5b6e4cf79f73e09427cee7ef: Generated new TSK 0
I20260925 02:56:53.775629 1829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:53.779832 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 1740
2026-09-25T02:56:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:53 dist-test-slave-252r krb5kdc[1842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:53 dist-test-slave-252r krb5kdc[1842](info): set up 2 sockets
Sep 25 02:56:53 dist-test-slave-252r krb5kdc[1842](info): commencing operation
krb5kdc: starting...
W20260925 02:56:54.748395 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:56:54 dist-test-slave-252r krb5kdc[1842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:56:54.859031 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45015
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:54.961565 1855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:54.961815 1855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:54.961922 1855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:54.965682 1855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:54.965757 1855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:54.965795 1855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:54.965852 1855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:54.965893 1855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:54.971060 1855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:54.971953 1855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:54.972868 1855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:54.980690 1863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:54.980777 1855 server_base.cc:1034] running on GCE node
W20260925 02:56:54.980695 1860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:54.980687 1861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:54.981515 1855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:54.982532 1855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:54.983673 1855 hybrid_clock.cc:648] HybridClock initialized: now 1790305014983657 us; error 26 us; skew 500 ppm
Sep 25 02:56:54 dist-test-slave-252r krb5kdc[1842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:54.986296 1855 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:56:54.987092 1855 webserver.cc:460] Webserver started at http://127.0.0.1:45103/ using document root <none> and password file <none>
I20260925 02:56:54.987556 1855 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:54.987608 1855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:54.987751 1855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:54.989602 1855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "cf2e2bc9088346ce87679d6fb3a669e9"
format_stamp: "Formatted at 2026-09-25 02:56:54 on dist-test-slave-252r"
server_key: "8adc01c2f881d455e80c6e886e8714e0"
server_key_iv: "e1c68976721887a21154e31f80fa06f8"
server_key_version: "encryptionkey@0"
I20260925 02:56:54.990072 1855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "cf2e2bc9088346ce87679d6fb3a669e9"
format_stamp: "Formatted at 2026-09-25 02:56:54 on dist-test-slave-252r"
server_key: "8adc01c2f881d455e80c6e886e8714e0"
server_key_iv: "e1c68976721887a21154e31f80fa06f8"
server_key_version: "encryptionkey@0"
I20260925 02:56:54.993436 1855 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:56:54.995711 1870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:54.996874 1855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:56:54.997005 1855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "cf2e2bc9088346ce87679d6fb3a669e9"
format_stamp: "Formatted at 2026-09-25 02:56:54 on dist-test-slave-252r"
server_key: "8adc01c2f881d455e80c6e886e8714e0"
server_key_iv: "e1c68976721887a21154e31f80fa06f8"
server_key_version: "encryptionkey@0"
I20260925 02:56:54.997108 1855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:55.010145 1855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:55.012740 1855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:55.012924 1855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:55.019260 1855 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45015
I20260925 02:56:55.019289 1922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45015 every 8 connection(s)
I20260925 02:56:55.020117 1855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:55.022639 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:55.024082 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 1855
I20260925 02:56:55.024251 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:56:55.024547 13179 external_mini_cluster.cc:1428] Setting key a0f62be8d2abfe7fc22644a244ad3eca
I20260925 02:56:55.028173 1923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9: Bootstrap starting.
I20260925 02:56:55.030076 1923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:55.030779 1923 log.cc:824] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:56:55 dist-test-slave-252r krb5kdc[1842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:56:55.032689 1923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9: No bootstrap required, opened a new log
I20260925 02:56:55.036674 1923 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2e2bc9088346ce87679d6fb3a669e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45015 } }
I20260925 02:56:55.036854 1923 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:55.036918 1923 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2e2bc9088346ce87679d6fb3a669e9, State: Initialized, Role: FOLLOWER
I20260925 02:56:55.037283 1923 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2e2bc9088346ce87679d6fb3a669e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45015 } }
I20260925 02:56:55.037415 1923 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:55.037494 1923 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:55.037525 1926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:55.025851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:54699 (local address 127.0.0.1:45015)
0925 02:56:55.026229 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:55.026242 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:55.026273 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:56:55.026830 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:55.026845 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:55.027145 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:55.027440 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:55.028197 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:55.029081 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:55.029931 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:55.030431 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:55.030625 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:55.033243 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:55.033247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:55.033257 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:55.033279 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:55.034735 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:55.035266 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:55.035270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:55.035273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:55.035368 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:55.035687 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:55.035689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:55.035690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:55.036011 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:55.036158 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:55.036460 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 02:56:55.036685 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":127,"threads_started":1}
I20260925 02:56:55.037597 1923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:55.038338 1923 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2e2bc9088346ce87679d6fb3a669e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45015 } }
I20260925 02:56:55.038659 1923 leader_election.cc:302] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf2e2bc9088346ce87679d6fb3a669e9; no voters:
I20260925 02:56:55.038954 1923 leader_election.cc:288] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:55.039040 1928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:55.039227 1928 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9 [term 1 LEADER]: Becoming Leader. State: Replica: cf2e2bc9088346ce87679d6fb3a669e9, State: Running, Role: LEADER
I20260925 02:56:55.039558 1928 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2e2bc9088346ce87679d6fb3a669e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45015 } }
I20260925 02:56:55.040076 1923 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:55.040328 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:56:55.042732 1930 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf2e2bc9088346ce87679d6fb3a669e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2e2bc9088346ce87679d6fb3a669e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45015 } } }
I20260925 02:56:55.042693 1929 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf2e2bc9088346ce87679d6fb3a669e9. Latest consensus state: current_term: 1 leader_uuid: "cf2e2bc9088346ce87679d6fb3a669e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2e2bc9088346ce87679d6fb3a669e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45015 } } }
I20260925 02:56:55.042900 1929 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:55.042900 1930 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9 [sys.catalog]: This master's current role is: LEADER
W20260925 02:56:55.045282 1948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:56:55.045418 1948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:56:55.045567 1950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:55.046195 1950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:55.050127 1926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:55.042715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:52999 (local address 127.0.0.1:45015)
0925 02:56:55.042800 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:55.042804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:55.043025 (+ 221us) server_negotiation.cc:407] Connection header received
0925 02:56:55.043137 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:55.043140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:55.043184 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:55.043256 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:55.043876 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:55.044409 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:55.045223 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:55.045492 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:55.045613 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:55.047049 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:55.047053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:55.047055 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:55.047076 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:55.048461 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:55.048867 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:55.048870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:55.048871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:55.048943 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:55.049270 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:55.049273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:55.049274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:55.049452 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:55.049583 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:55.049905 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 02:56:55.049998 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 02:56:55.051049 1950 catalog_manager.cc:1347] Generated new cluster ID: f5c41b10ed284f99b8e0a7a807c26957
I20260925 02:56:55.051097 1950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:55.066740 1950 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:55.067463 1950 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:55.078378 1950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf2e2bc9088346ce87679d6fb3a669e9: Generated new TSK 0
I20260925 02:56:55.078977 1950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:55.083280 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 1855
W20260925 02:56:55.088721 1840 connection.cc:535] client connection to 127.0.0.1:45015 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45015): Connection reset by peer (error 104)
2026-09-25T02:56:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:55 dist-test-slave-252r krb5kdc[1955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:55 dist-test-slave-252r krb5kdc[1955](info): set up 2 sockets
Sep 25 02:56:55 dist-test-slave-252r krb5kdc[1955](info): commencing operation
krb5kdc: starting...
W20260925 02:56:56.063930 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:56:56 dist-test-slave-252r krb5kdc[1955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:56:56.173261 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:56.272047 1968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:56.272248 1968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:56.272282 1968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:56.275735 1968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:56.275779 1968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:56.275796 1968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:56.275811 1968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:56.275826 1968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:56.280519 1968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:56.281315 1968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:56.282135 1968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:56.289776 1976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:56.289810 1974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:56.289870 1968 server_base.cc:1034] running on GCE node
W20260925 02:56:56.289788 1973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:56.290668 1968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:56.291646 1968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:56.292840 1968 hybrid_clock.cc:648] HybridClock initialized: now 1790305016292828 us; error 33 us; skew 500 ppm
Sep 25 02:56:56 dist-test-slave-252r krb5kdc[1955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:56.300381 1968 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:56:56.301229 1968 webserver.cc:460] Webserver started at http://127.0.0.1:41433/ using document root <none> and password file <none>
I20260925 02:56:56.301743 1968 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:56.301828 1968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:56.302039 1968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:56.303905 1968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "423b9a1872c84c21b11c1a208edacc28"
format_stamp: "Formatted at 2026-09-25 02:56:56 on dist-test-slave-252r"
server_key: "fb15581dbf978482166f98f819c02f2f"
server_key_iv: "669b8c9e9374bdd009c642afc5cd2a45"
server_key_version: "encryptionkey@0"
I20260925 02:56:56.304481 1968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "423b9a1872c84c21b11c1a208edacc28"
format_stamp: "Formatted at 2026-09-25 02:56:56 on dist-test-slave-252r"
server_key: "fb15581dbf978482166f98f819c02f2f"
server_key_iv: "669b8c9e9374bdd009c642afc5cd2a45"
server_key_version: "encryptionkey@0"
I20260925 02:56:56.307869 1968 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:56:56.309984 1983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:56.311033 1968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 02:56:56.311162 1968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "423b9a1872c84c21b11c1a208edacc28"
format_stamp: "Formatted at 2026-09-25 02:56:56 on dist-test-slave-252r"
server_key: "fb15581dbf978482166f98f819c02f2f"
server_key_iv: "669b8c9e9374bdd009c642afc5cd2a45"
server_key_version: "encryptionkey@0"
I20260925 02:56:56.311259 1968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:56.323877 1968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:56.326825 1968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:56.327013 1968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:56.333557 1968 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36955
I20260925 02:56:56.333577 2036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36955 every 8 connection(s)
I20260925 02:56:56.334339 1968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:56.336879 2037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:56.338905 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 1968
I20260925 02:56:56.339028 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:56:56.339314 13179 external_mini_cluster.cc:1428] Setting key d13f723795bdaea83c45b2d233ea0505
I20260925 02:56:56.342640 2037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28: Bootstrap starting.
I20260925 02:56:56.344959 2037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:56.345790 2037 log.cc:824] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:56.347924 2037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28: No bootstrap required, opened a new log
I20260925 02:56:56.348577 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:56.340625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:51687 (local address 127.0.0.1:36955)
0925 02:56:56.341036 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:56.341055 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:56.341094 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:56:56.341884 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:56.341907 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:56.342287 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:56.342618 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:56.342999 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:56.344322 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:56.345249 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:56.345912 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:56.346066 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:56.346791 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:56.346796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:56.346809 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:56.347237 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:56.347298 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:56.347326 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:56.347448 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:56.347557 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 02:56:56.347699 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":130,"threads_started":1}
I20260925 02:56:56.351343 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:56:56.351884 2037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423b9a1872c84c21b11c1a208edacc28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } }
I20260925 02:56:56.352069 2037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:56.352124 2037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 423b9a1872c84c21b11c1a208edacc28, State: Initialized, Role: FOLLOWER
I20260925 02:56:56.352576 2037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423b9a1872c84c21b11c1a208edacc28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } }
I20260925 02:56:56.352758 2037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:56.352846 2037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:56.352959 2037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:56.354163 2037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423b9a1872c84c21b11c1a208edacc28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } }
I20260925 02:56:56.354511 2037 leader_election.cc:302] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 423b9a1872c84c21b11c1a208edacc28; no voters:
I20260925 02:56:56.354815 2037 leader_election.cc:288] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:56.354926 2047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:56.355219 2047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28 [term 1 LEADER]: Becoming Leader. State: Replica: 423b9a1872c84c21b11c1a208edacc28, State: Running, Role: LEADER
I20260925 02:56:56.355556 2047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423b9a1872c84c21b11c1a208edacc28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } }
I20260925 02:56:56.355933 2037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:56.356808 2048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 423b9a1872c84c21b11c1a208edacc28. Latest consensus state: current_term: 1 leader_uuid: "423b9a1872c84c21b11c1a208edacc28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423b9a1872c84c21b11c1a208edacc28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } } }
I20260925 02:56:56.357005 2048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:56.357488 2056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:56.357494 2049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "423b9a1872c84c21b11c1a208edacc28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423b9a1872c84c21b11c1a208edacc28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } } }
I20260925 02:56:56.357708 2049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:56.358002 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:56.353443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:52091 (local address 127.0.0.1:36955)
0925 02:56:56.353540 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:56.353543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:56.353758 (+ 215us) server_negotiation.cc:407] Connection header received
0925 02:56:56.353825 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:56.353831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:56.353885 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:56.353964 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:56.354196 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:56.354923 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:56.356058 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:56.356427 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:56.356527 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:56.357445 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:56.357449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:56.357451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:56.357462 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:56.357488 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:56.357501 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:56.357585 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:56.357868 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 02:56:56.357898 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:56:56.360374 2056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:56.365175 2056 catalog_manager.cc:1347] Generated new cluster ID: 1a11c10e54624c19ac696ffcb6a76faa
I20260925 02:56:56.365242 2056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:56.386624 2056 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:56.387528 2056 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:56.394681 2056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 423b9a1872c84c21b11c1a208edacc28: Generated new TSK 0
I20260925 02:56:56.395361 2056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:56.400827 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 1968
W20260925 02:56:56.406347 1953 connection.cc:535] client connection to 127.0.0.1:36955 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36955): Connection reset by peer (error 104)
2026-09-25T02:56:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:56 dist-test-slave-252r krb5kdc[2068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:56 dist-test-slave-252r krb5kdc[2068](info): set up 2 sockets
Sep 25 02:56:56 dist-test-slave-252r krb5kdc[2068](info): commencing operation
krb5kdc: starting...
W20260925 02:56:57.360190 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:56:57 dist-test-slave-252r krb5kdc[2068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:56:57.468773 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36575
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:57.570269 2081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:57.570508 2081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:57.570547 2081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:57.574169 2081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:57.574213 2081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:57.574229 2081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:57.574244 2081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:57.574257 2081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:57.578939 2081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:57.579792 2081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:57.580695 2081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:57.588354 2086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:57.588425 2087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:57.588622 2081 server_base.cc:1034] running on GCE node
W20260925 02:56:57.588354 2089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:57.589383 2081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:57.590423 2081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:57.591602 2081 hybrid_clock.cc:648] HybridClock initialized: now 1790305017591579 us; error 41 us; skew 500 ppm
Sep 25 02:56:57 dist-test-slave-252r krb5kdc[2068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:57.594040 2081 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:56:57.594851 2081 webserver.cc:460] Webserver started at http://127.0.0.1:38797/ using document root <none> and password file <none>
I20260925 02:56:57.595396 2081 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:57.595484 2081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:57.595724 2081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:57.597656 2081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "6fe9de4ee622487c85271e1f0c33eccb"
format_stamp: "Formatted at 2026-09-25 02:56:57 on dist-test-slave-252r"
server_key: "ec82391732245d67fd19eed5496fc4ed"
server_key_iv: "cd0a3aa0e42b9e237f79369ab77f3b1e"
server_key_version: "encryptionkey@0"
I20260925 02:56:57.598169 2081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "6fe9de4ee622487c85271e1f0c33eccb"
format_stamp: "Formatted at 2026-09-25 02:56:57 on dist-test-slave-252r"
server_key: "ec82391732245d67fd19eed5496fc4ed"
server_key_iv: "cd0a3aa0e42b9e237f79369ab77f3b1e"
server_key_version: "encryptionkey@0"
I20260925 02:56:57.601580 2081 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:56:57.603804 2096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:57.604983 2081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:56:57.605116 2081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "6fe9de4ee622487c85271e1f0c33eccb"
format_stamp: "Formatted at 2026-09-25 02:56:57 on dist-test-slave-252r"
server_key: "ec82391732245d67fd19eed5496fc4ed"
server_key_iv: "cd0a3aa0e42b9e237f79369ab77f3b1e"
server_key_version: "encryptionkey@0"
I20260925 02:56:57.605217 2081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:57.609288 2081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:57.611809 2081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:57.611995 2081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:57.618889 2081 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36575
I20260925 02:56:57.618916 2149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36575 every 8 connection(s)
I20260925 02:56:57.619738 2081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:57.622403 2150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:57.624161 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 2081
I20260925 02:56:57.624301 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:56:57.624579 13179 external_mini_cluster.cc:1428] Setting key c6a8133d180e774dd733c4ff6345eec7
I20260925 02:56:57.628398 2150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb: Bootstrap starting.
I20260925 02:56:57.630179 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:57.625913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:45373 (local address 127.0.0.1:36575)
0925 02:56:57.626325 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:57.626342 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:57.626380 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:56:57.627317 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:57.627340 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:57.627674 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:57.628049 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:57.628842 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:57.628848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:57.628861 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:57.628873 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:57.628940 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:57.628966 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:57.629095 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:57.629158 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 02:56:57.629297 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":114,"threads_started":1}
I20260925 02:56:57.630993 2150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:57.631747 2150 log.cc:824] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:57.632846 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:56:57.633682 2150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb: No bootstrap required, opened a new log
I20260925 02:56:57.636549 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:57.634840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:49353 (local address 127.0.0.1:36575)
0925 02:56:57.634978 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:57.634982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:57.635190 (+ 208us) server_negotiation.cc:407] Connection header received
0925 02:56:57.635271 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:57.635274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:57.635307 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:57.635382 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:57.636105 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:57.636109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:57.636112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:56:57.636123 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:57.636150 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:56:57.636165 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:57.636260 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:57.636398 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 02:56:57.636437 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:56:57.637970 2150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fe9de4ee622487c85271e1f0c33eccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36575 } }
I20260925 02:56:57.638180 2150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:57.638218 2150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fe9de4ee622487c85271e1f0c33eccb, State: Initialized, Role: FOLLOWER
I20260925 02:56:57.638574 2150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fe9de4ee622487c85271e1f0c33eccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36575 } }
I20260925 02:56:57.638682 2150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:57.638751 2150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:57.638849 2150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:56:57.639757 2150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fe9de4ee622487c85271e1f0c33eccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36575 } }
I20260925 02:56:57.640136 2150 leader_election.cc:302] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fe9de4ee622487c85271e1f0c33eccb; no voters:
I20260925 02:56:57.640571 2150 leader_election.cc:288] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:57.640681 2160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:57.640954 2160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb [term 1 LEADER]: Becoming Leader. State: Replica: 6fe9de4ee622487c85271e1f0c33eccb, State: Running, Role: LEADER
I20260925 02:56:57.641264 2160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fe9de4ee622487c85271e1f0c33eccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36575 } }
I20260925 02:56:57.641944 2150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:57.642943 2162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fe9de4ee622487c85271e1f0c33eccb. Latest consensus state: current_term: 1 leader_uuid: "6fe9de4ee622487c85271e1f0c33eccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fe9de4ee622487c85271e1f0c33eccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36575 } } }
I20260925 02:56:57.643069 2162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:57.643389 2161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fe9de4ee622487c85271e1f0c33eccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fe9de4ee622487c85271e1f0c33eccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36575 } } }
I20260925 02:56:57.643478 2161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:57.643579 2169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:57.646273 2169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:57.651366 2169 catalog_manager.cc:1347] Generated new cluster ID: 30f4b2e0f93748a7a80b229da74a280b
I20260925 02:56:57.651438 2169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:57.659639 2169 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:57.659935 2169 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:57.671985 2169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fe9de4ee622487c85271e1f0c33eccb: Generated new TSK 0
I20260925 02:56:57.672770 2169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:57.681841 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 2081
2026-09-25T02:56:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:57 dist-test-slave-252r krb5kdc[2181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:57 dist-test-slave-252r krb5kdc[2181](info): set up 2 sockets
Sep 25 02:56:57 dist-test-slave-252r krb5kdc[2181](info): commencing operation
krb5kdc: starting...
W20260925 02:56:58.644035 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:56:58 dist-test-slave-252r krb5kdc[2181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:56:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:56:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:56:58.756798 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:56:58.874616 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:56:58.874845 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:56:58.874883 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:56:58.878608 2194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:56:58.878650 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:56:58.878667 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:56:58.878681 2194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:56:58.878695 2194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:56:58.883344 2194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36267
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:56:58.884171 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:56:58.885120 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:56:58.892935 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:58.892927 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:56:58.892936 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:56:58.893296 2194 server_base.cc:1034] running on GCE node
I20260925 02:56:58.893715 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:56:58.894665 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:56:58.895819 2194 hybrid_clock.cc:648] HybridClock initialized: now 1790305018895797 us; error 33 us; skew 500 ppm
Sep 25 02:56:58 dist-test-slave-252r krb5kdc[2181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:56:58.898432 2194 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:56:58.899236 2194 webserver.cc:460] Webserver started at http://127.0.0.1:39799/ using document root <none> and password file <none>
I20260925 02:56:58.899684 2194 fs_manager.cc:359] Metadata directory not provided
I20260925 02:56:58.899734 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:56:58.899874 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:56:58.901834 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "537acb4fc9624398a868bf3cb8ff6796"
format_stamp: "Formatted at 2026-09-25 02:56:58 on dist-test-slave-252r"
server_key: "68c8528129892354861c89ce667912e5"
server_key_iv: "dd54a1dffb9b6acf28f2c2277fb83dab"
server_key_version: "encryptionkey@0"
I20260925 02:56:58.902288 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "537acb4fc9624398a868bf3cb8ff6796"
format_stamp: "Formatted at 2026-09-25 02:56:58 on dist-test-slave-252r"
server_key: "68c8528129892354861c89ce667912e5"
server_key_iv: "dd54a1dffb9b6acf28f2c2277fb83dab"
server_key_version: "encryptionkey@0"
I20260925 02:56:58.905493 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:56:58.907497 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:56:58.908605 2194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:56:58.908735 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "537acb4fc9624398a868bf3cb8ff6796"
format_stamp: "Formatted at 2026-09-25 02:56:58 on dist-test-slave-252r"
server_key: "68c8528129892354861c89ce667912e5"
server_key_iv: "dd54a1dffb9b6acf28f2c2277fb83dab"
server_key_version: "encryptionkey@0"
I20260925 02:56:58.908815 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:56:58.923153 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:56:58.926097 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:56:58.926296 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:56:58.932740 2194 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36267
I20260925 02:56:58.932765 2262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36267 every 8 connection(s)
I20260925 02:56:58.933538 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:56:58.935940 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:56:58.940959 2263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796: Bootstrap starting.
I20260925 02:56:58.942209 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 2194
I20260925 02:56:58.942305 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:56:58.942613 13179 external_mini_cluster.cc:1428] Setting key 42e278ab03a3097eac36a3e44c5338cf
I20260925 02:56:58.943301 2263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796: Neither blocks nor log segments found. Creating new log.
I20260925 02:56:58.944115 2263 log.cc:824] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796: Log is configured to *not* fsync() on all Append() calls
I20260925 02:56:58.946084 2263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796: No bootstrap required, opened a new log
I20260925 02:56:58.949854 2263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537acb4fc9624398a868bf3cb8ff6796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36267 } }
I20260925 02:56:58.950125 2263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:56:58.950186 2263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 537acb4fc9624398a868bf3cb8ff6796, State: Initialized, Role: FOLLOWER
I20260925 02:56:58.950670 2263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537acb4fc9624398a868bf3cb8ff6796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36267 } }
I20260925 02:56:58.950801 2263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:56:58.950891 2263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:56:58.951015 2263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 02:56:58 dist-test-slave-252r krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:56:58.951939 2263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537acb4fc9624398a868bf3cb8ff6796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36267 } }
I20260925 02:56:58.952275 2263 leader_election.cc:302] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 537acb4fc9624398a868bf3cb8ff6796; no voters:
I20260925 02:56:58.952585 2263 leader_election.cc:288] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:56:58.952641 2268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:56:58.952824 2268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796 [term 1 LEADER]: Becoming Leader. State: Replica: 537acb4fc9624398a868bf3cb8ff6796, State: Running, Role: LEADER
I20260925 02:56:58.953140 2268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537acb4fc9624398a868bf3cb8ff6796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36267 } }
I20260925 02:56:58.953701 2263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:56:58.954658 2269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 537acb4fc9624398a868bf3cb8ff6796. Latest consensus state: current_term: 1 leader_uuid: "537acb4fc9624398a868bf3cb8ff6796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537acb4fc9624398a868bf3cb8ff6796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36267 } } }
I20260925 02:56:58.954780 2269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:58.955140 2270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "537acb4fc9624398a868bf3cb8ff6796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537acb4fc9624398a868bf3cb8ff6796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36267 } } }
I20260925 02:56:58.955286 2270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796 [sys.catalog]: This master's current role is: LEADER
I20260925 02:56:58.957150 2266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:58.943639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:48845 (local address 127.0.0.1:36267)
0925 02:56:58.944000 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:58.944015 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:58.944046 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:56:58.944844 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:58.944864 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:58.945179 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:58.945442 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:58.946332 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:58.947185 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:58.948334 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:58.948750 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:58.948875 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:58.951778 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:58.951783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:58.951795 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:58.951820 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:58.953574 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:58.954136 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:58.954158 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:58.954165 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:58.954298 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:58.954738 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:58.954741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:58.954743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:58.955278 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:58.955490 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:58.956377 (+ 887us) server_negotiation.cc:299] Negotiation successful
0925 02:56:58.956532 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":114,"threads_started":1}
W20260925 02:56:58.958433 2283 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:56:58.958516 2283 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:56:58.958575 2277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:56:58.959424 2277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:56:58.959826 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:56:58.964857 2277 catalog_manager.cc:1347] Generated new cluster ID: 39fd562e9973465b8992acc4eb531658
I20260925 02:56:58.964918 2277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:56:58.970333 2266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:56:58.962997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:57385 (local address 127.0.0.1:36267)
0925 02:56:58.963105 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 02:56:58.963109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:56:58.963122 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:56:58.963152 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:56:58.963155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:56:58.963193 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:56:58.963271 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:56:58.964058 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:58.964773 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:58.965665 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:56:58.965923 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:56:58.966040 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:56:58.967290 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:56:58.967293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:56:58.967296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:56:58.967316 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:56:58.968613 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:58.969103 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:58.969106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:58.969108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:58.969183 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:56:58.969506 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:56:58.969509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:56:58.969511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:56:58.969665 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:56:58.969812 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 02:56:58.970083 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 02:56:58.970166 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:56:58.984745 2277 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:56:58.985569 2277 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:56:58.991209 2277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 537acb4fc9624398a868bf3cb8ff6796: Generated new TSK 0
I20260925 02:56:58.991781 2277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:56:58.995008 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 2194
W20260925 02:56:59.000540 2179 connection.cc:535] client connection to 127.0.0.1:36267 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36267): Connection reset by peer (error 104)
2026-09-25T02:56:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:56:59 dist-test-slave-252r krb5kdc[2294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:56:59 dist-test-slave-252r krb5kdc[2294](info): set up 2 sockets
Sep 25 02:56:59 dist-test-slave-252r krb5kdc[2294](info): commencing operation
krb5kdc: starting...
W20260925 02:56:59.968544 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:57:00 dist-test-slave-252r krb5kdc[2294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:57:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:57:00.079192 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:00.180430 2307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:00.180635 2307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:00.180689 2307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:00.184592 2307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:00.184639 2307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:00.184657 2307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:00.184674 2307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:00.184724 2307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:00.189592 2307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37477
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:57:00.190502 2307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:00.191425 2307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:00.199349 2312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:00.199347 2313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:00.199483 2307 server_base.cc:1034] running on GCE node
W20260925 02:57:00.199329 2315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:00.200026 2307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:00.201011 2307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:00.202173 2307 hybrid_clock.cc:648] HybridClock initialized: now 1790305020202155 us; error 28 us; skew 500 ppm
Sep 25 02:57:00 dist-test-slave-252r krb5kdc[2294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:00.204748 2307 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:57:00.205519 2307 webserver.cc:460] Webserver started at http://127.0.0.1:43413/ using document root <none> and password file <none>
I20260925 02:57:00.206017 2307 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:00.206073 2307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:00.206274 2307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:00.208045 2307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "0cf6d07836fe46f4b71e668803927686"
format_stamp: "Formatted at 2026-09-25 02:57:00 on dist-test-slave-252r"
server_key: "79a13d143d44b18302fedb6c7ec0cc23"
server_key_iv: "144888ab0620c2f334de6b0f2fca5bee"
server_key_version: "encryptionkey@0"
I20260925 02:57:00.208636 2307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "0cf6d07836fe46f4b71e668803927686"
format_stamp: "Formatted at 2026-09-25 02:57:00 on dist-test-slave-252r"
server_key: "79a13d143d44b18302fedb6c7ec0cc23"
server_key_iv: "144888ab0620c2f334de6b0f2fca5bee"
server_key_version: "encryptionkey@0"
I20260925 02:57:00.211934 2307 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:57:00.214186 2322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:00.215287 2307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:57:00.215405 2307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "0cf6d07836fe46f4b71e668803927686"
format_stamp: "Formatted at 2026-09-25 02:57:00 on dist-test-slave-252r"
server_key: "79a13d143d44b18302fedb6c7ec0cc23"
server_key_iv: "144888ab0620c2f334de6b0f2fca5bee"
server_key_version: "encryptionkey@0"
I20260925 02:57:00.215497 2307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:00.235416 2307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:00.237874 2307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:00.238063 2307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:00.245123 2307 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37477
I20260925 02:57:00.245168 2374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37477 every 8 connection(s)
I20260925 02:57:00.246039 2307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:00.248683 2375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:00.253883 2375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686: Bootstrap starting.
I20260925 02:57:00.255039 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 2307
I20260925 02:57:00.255230 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:57:00.255509 13179 external_mini_cluster.cc:1428] Setting key 538b173e176e9ba928d4f14654eae609
I20260925 02:57:00.256610 2375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:00.257442 2375 log.cc:824] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:00.259469 2375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686: No bootstrap required, opened a new log
I20260925 02:57:00.263392 2375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf6d07836fe46f4b71e668803927686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37477 } }
I20260925 02:57:00.263664 2375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:00.263783 2375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf6d07836fe46f4b71e668803927686, State: Initialized, Role: FOLLOWER
I20260925 02:57:00.264246 2375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf6d07836fe46f4b71e668803927686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37477 } }
I20260925 02:57:00.264386 2375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:00.264446 2375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:00.264528 2375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:00.264813 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:00.256644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:55083 (local address 127.0.0.1:37477)
0925 02:57:00.257081 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:00.257098 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:00.257136 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:57:00.257941 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:00.257960 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:00.258328 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:00.258681 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:00.259072 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:00.260432 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:00.261290 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:00.261988 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:00.262176 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:00.262919 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:00.262924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:00.262940 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:00.263367 (+ 427us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:00.263439 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:00.263485 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:00.263603 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:00.263793 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 02:57:00.263917 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":139,"threads_started":1}
I20260925 02:57:00.265522 2375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf6d07836fe46f4b71e668803927686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37477 } }
I20260925 02:57:00.265849 2375 leader_election.cc:302] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cf6d07836fe46f4b71e668803927686; no voters:
I20260925 02:57:00.266181 2375 leader_election.cc:288] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:00.266268 2380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:00.266489 2380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686 [term 1 LEADER]: Becoming Leader. State: Replica: 0cf6d07836fe46f4b71e668803927686, State: Running, Role: LEADER
I20260925 02:57:00.266752 2380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf6d07836fe46f4b71e668803927686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37477 } }
I20260925 02:57:00.267414 2375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:00.267843 2380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf6d07836fe46f4b71e668803927686. Latest consensus state: current_term: 1 leader_uuid: "0cf6d07836fe46f4b71e668803927686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf6d07836fe46f4b71e668803927686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37477 } } }
I20260925 02:57:00.267937 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:57:00.267843 2382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cf6d07836fe46f4b71e668803927686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf6d07836fe46f4b71e668803927686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37477 } } }
I20260925 02:57:00.268059 2382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:00.268059 2380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:00.268538 2386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:00.271430 2386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:00.275632 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:00.270159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:55453 (local address 127.0.0.1:37477)
0925 02:57:00.270272 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:00.270276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:00.270511 (+ 235us) server_negotiation.cc:407] Connection header received
0925 02:57:00.270671 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:00.270673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:00.270714 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:00.270788 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:00.271150 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:00.272403 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:00.273549 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:00.273879 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:00.274001 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:00.274974 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:00.274977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:00.274980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:00.274991 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:00.275032 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:00.275045 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:00.275128 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:00.275454 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 02:57:00.275504 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:57:00.276315 2386 catalog_manager.cc:1347] Generated new cluster ID: 7eb1016175b3427a8b967d1dd6b17fb8
I20260925 02:57:00.276424 2386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:00.291152 2386 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:00.292061 2386 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:00.299002 2386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cf6d07836fe46f4b71e668803927686: Generated new TSK 0
I20260925 02:57:00.299575 2386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:57:00.304379 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 2307
W20260925 02:57:00.309860 2292 connection.cc:535] client connection to 127.0.0.1:37477 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37477): Connection reset by peer (error 104)
2026-09-25T02:57:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:57:00 dist-test-slave-252r krb5kdc[2406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:57:00 dist-test-slave-252r krb5kdc[2406](info): set up 2 sockets
Sep 25 02:57:00 dist-test-slave-252r krb5kdc[2406](info): commencing operation
krb5kdc: starting...
W20260925 02:57:01.320611 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:57:01 dist-test-slave-252r krb5kdc[2406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:57:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:57:01.432204 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:01.534420 2419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:01.534685 2419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:01.534768 2419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:01.538617 2419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:01.538690 2419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:01.538727 2419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:01.538796 2419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:01.538831 2419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:01.543708 2419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:57:01.544644 2419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:01.545593 2419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:57:01.553514 2419 server_base.cc:1034] running on GCE node
W20260925 02:57:01.553460 2424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:01.553479 2425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:01.553448 2427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:01.554170 2419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:01.555194 2419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:01.556396 2419 hybrid_clock.cc:648] HybridClock initialized: now 1790305021556361 us; error 38 us; skew 500 ppm
Sep 25 02:57:01 dist-test-slave-252r krb5kdc[2406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:01.563810 2419 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:57:01.564636 2419 webserver.cc:460] Webserver started at http://127.0.0.1:46839/ using document root <none> and password file <none>
I20260925 02:57:01.565150 2419 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:01.565238 2419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:01.565502 2419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:01.567473 2419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "ca26b27a450e43098ed7c5da638b444a"
format_stamp: "Formatted at 2026-09-25 02:57:01 on dist-test-slave-252r"
server_key: "d3feef9b188ea52b3956dfd061cba624"
server_key_iv: "bb7fe926164098bbc12e2c263a5b7e67"
server_key_version: "encryptionkey@0"
I20260925 02:57:01.568018 2419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "ca26b27a450e43098ed7c5da638b444a"
format_stamp: "Formatted at 2026-09-25 02:57:01 on dist-test-slave-252r"
server_key: "d3feef9b188ea52b3956dfd061cba624"
server_key_iv: "bb7fe926164098bbc12e2c263a5b7e67"
server_key_version: "encryptionkey@0"
I20260925 02:57:01.571553 2419 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:57:01.573877 2435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:01.575050 2419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 02:57:01.575179 2419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "ca26b27a450e43098ed7c5da638b444a"
format_stamp: "Formatted at 2026-09-25 02:57:01 on dist-test-slave-252r"
server_key: "d3feef9b188ea52b3956dfd061cba624"
server_key_iv: "bb7fe926164098bbc12e2c263a5b7e67"
server_key_version: "encryptionkey@0"
I20260925 02:57:01.575307 2419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:01.579638 2419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:01.582223 2419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:01.582410 2419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:01.588956 2419 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38973
I20260925 02:57:01.588992 2487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38973 every 8 connection(s)
I20260925 02:57:01.589785 2419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:01.592301 2488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:01.597093 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 2419
I20260925 02:57:01.597249 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:57:01.597438 2488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a: Bootstrap starting.
I20260925 02:57:01.597550 13179 external_mini_cluster.cc:1428] Setting key f9d4c5b132a48f01137cf5fa4be18c0e
I20260925 02:57:01.599985 2488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:01.600848 2488 log.cc:824] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:01.602684 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:01.598810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:43391 (local address 127.0.0.1:38973)
0925 02:57:01.599251 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:01.599268 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:01.599309 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:57:01.600135 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:01.600163 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:01.600522 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:01.600889 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:01.601611 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:01.601616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:01.601626 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:01.601636 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:01.601685 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:01.601702 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:01.601812 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:01.601934 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 02:57:01.602086 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":115,"threads_started":1}
I20260925 02:57:01.602705 2488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a: No bootstrap required, opened a new log
I20260925 02:57:01.605125 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:57:01.606946 2488 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca26b27a450e43098ed7c5da638b444a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38973 } }
I20260925 02:57:01.607193 2488 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:01.607254 2488 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca26b27a450e43098ed7c5da638b444a, State: Initialized, Role: FOLLOWER
I20260925 02:57:01.607689 2488 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca26b27a450e43098ed7c5da638b444a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38973 } }
I20260925 02:57:01.607862 2488 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:01.607930 2488 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:01.608042 2488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:01.608659 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:01.607010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:55571 (local address 127.0.0.1:38973)
0925 02:57:01.607127 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:01.607131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:01.607356 (+ 225us) server_negotiation.cc:407] Connection header received
0925 02:57:01.607512 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:01.607515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:01.607545 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:01.607608 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:01.608311 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:01.608314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:01.608317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:01.608332 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:01.608352 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:01.608361 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:01.608424 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:01.608526 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 02:57:01.608551 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:57:01.609050 2488 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca26b27a450e43098ed7c5da638b444a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38973 } }
I20260925 02:57:01.609364 2488 leader_election.cc:302] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca26b27a450e43098ed7c5da638b444a; no voters:
I20260925 02:57:01.609638 2488 leader_election.cc:288] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:01.609740 2498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:01.609951 2498 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a [term 1 LEADER]: Becoming Leader. State: Replica: ca26b27a450e43098ed7c5da638b444a, State: Running, Role: LEADER
I20260925 02:57:01.610249 2498 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca26b27a450e43098ed7c5da638b444a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38973 } }
I20260925 02:57:01.610754 2488 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:01.611702 2499 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca26b27a450e43098ed7c5da638b444a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca26b27a450e43098ed7c5da638b444a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38973 } } }
I20260925 02:57:01.611850 2499 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:01.612144 2500 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca26b27a450e43098ed7c5da638b444a. Latest consensus state: current_term: 1 leader_uuid: "ca26b27a450e43098ed7c5da638b444a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca26b27a450e43098ed7c5da638b444a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38973 } } }
I20260925 02:57:01.612205 2507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:01.612329 2500 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:01.614750 2507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:01.619935 2507 catalog_manager.cc:1347] Generated new cluster ID: 362695918b4c4880818c0538f36f2a6a
I20260925 02:57:01.619993 2507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:01.641292 2507 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:01.641577 2507 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:01.649194 2507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca26b27a450e43098ed7c5da638b444a: Generated new TSK 0
I20260925 02:57:01.649757 2507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:57:01.652758 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 2419
2026-09-25T02:57:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:57:01 dist-test-slave-252r krb5kdc[2519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:57:01 dist-test-slave-252r krb5kdc[2519](info): set up 2 sockets
Sep 25 02:57:01 dist-test-slave-252r krb5kdc[2519](info): commencing operation
krb5kdc: starting...
W20260925 02:57:02.615818 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:57:02 dist-test-slave-252r krb5kdc[2519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:57:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:57:02.723569 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:36457
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:36457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:02.824498 2532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:02.824704 2532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:02.824762 2532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:02.828478 2532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:02.828524 2532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:02.828541 2532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:02.828557 2532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:02.828569 2532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:02.833017 2532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:36457
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:36457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:57:02.833832 2532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:02.834671 2532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:57:02.842367 2532 server_base.cc:1034] running on GCE node
W20260925 02:57:02.842305 2537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:02.842289 2538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:02.842319 2540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:02.843071 2532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:02.844023 2532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:02.845175 2532 hybrid_clock.cc:648] HybridClock initialized: now 1790305022845159 us; error 27 us; skew 500 ppm
Sep 25 02:57:02 dist-test-slave-252r krb5kdc[2519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305022, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:02.847512 2532 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:57:02.848312 2532 webserver.cc:460] Webserver started at http://127.12.222.254:38007/ using document root <none> and password file <none>
I20260925 02:57:02.848783 2532 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:02.848866 2532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:02.849068 2532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:02.850867 2532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "372626cabea24be6aa91ee9374bd77ec"
format_stamp: "Formatted at 2026-09-25 02:57:02 on dist-test-slave-252r"
server_key: "c60839240d06c232ec43e2985130590d"
server_key_iv: "03e8dec4062ebbded10ade243f7c8344"
server_key_version: "encryptionkey@0"
I20260925 02:57:02.851399 2532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "372626cabea24be6aa91ee9374bd77ec"
format_stamp: "Formatted at 2026-09-25 02:57:02 on dist-test-slave-252r"
server_key: "c60839240d06c232ec43e2985130590d"
server_key_iv: "03e8dec4062ebbded10ade243f7c8344"
server_key_version: "encryptionkey@0"
I20260925 02:57:02.854799 2532 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260925 02:57:02.857028 2547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:02.858057 2532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260925 02:57:02.858186 2532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "372626cabea24be6aa91ee9374bd77ec"
format_stamp: "Formatted at 2026-09-25 02:57:02 on dist-test-slave-252r"
server_key: "c60839240d06c232ec43e2985130590d"
server_key_iv: "03e8dec4062ebbded10ade243f7c8344"
server_key_version: "encryptionkey@0"
I20260925 02:57:02.858280 2532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:02.877905 2532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:02.880380 2532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:02.880554 2532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:02.886883 2532 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:36457
I20260925 02:57:02.886911 2599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:36457 every 8 connection(s)
I20260925 02:57:02.887688 2532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:02.888490 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 2532
I20260925 02:57:02.888640 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:57:02.888909 13179 external_mini_cluster.cc:1428] Setting key ec22130e272ce818c669c8b27b1a7327
I20260925 02:57:02.890530 2600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:02.895560 2600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec: Bootstrap starting.
Sep 25 02:57:02 dist-test-slave-252r krb5kdc[2519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:57:02.897704 2600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:02.898556 2600 log.cc:824] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:02.900190 2600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec: No bootstrap required, opened a new log
I20260925 02:57:02.901811 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:02.890109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37987 (local address 127.12.222.254:36457)
0925 02:57:02.890503 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:02.890516 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:02.890546 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:57:02.891151 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:02.891166 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:02.891467 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:02.891771 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:02.892538 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:02.893470 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:02.894322 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:02.894785 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:02.894944 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:02.897493 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:02.897497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:02.897508 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:02.897531 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:02.899113 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:02.899584 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:02.899589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:02.899593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:02.899690 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:02.900010 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:02.900013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:02.900014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:02.900468 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:02.900677 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:02.900988 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 02:57:02.901191 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":126,"threads_started":1}
I20260925 02:57:02.903529 2600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372626cabea24be6aa91ee9374bd77ec" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 36457 } }
I20260925 02:57:02.903767 2600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:02.903831 2600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372626cabea24be6aa91ee9374bd77ec, State: Initialized, Role: FOLLOWER
I20260925 02:57:02.904209 2600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372626cabea24be6aa91ee9374bd77ec" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 36457 } }
I20260925 02:57:02.904403 2600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:02.904474 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:57:02.904477 2600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:02.904562 2600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:02.905309 2600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372626cabea24be6aa91ee9374bd77ec" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 36457 } }
I20260925 02:57:02.905694 2600 leader_election.cc:302] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 372626cabea24be6aa91ee9374bd77ec; no voters:
I20260925 02:57:02.906055 2600 leader_election.cc:288] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:02.906127 2609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:02.907231 2600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:02.908376 2609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec [term 1 LEADER]: Becoming Leader. State: Replica: 372626cabea24be6aa91ee9374bd77ec, State: Running, Role: LEADER
I20260925 02:57:02.908643 2609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372626cabea24be6aa91ee9374bd77ec" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 36457 } }
I20260925 02:57:02.910049 2610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "372626cabea24be6aa91ee9374bd77ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372626cabea24be6aa91ee9374bd77ec" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 36457 } } }
I20260925 02:57:02.910171 2610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:02.910599 2619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:02.910563 2617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 372626cabea24be6aa91ee9374bd77ec. Latest consensus state: current_term: 1 leader_uuid: "372626cabea24be6aa91ee9374bd77ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372626cabea24be6aa91ee9374bd77ec" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 36457 } } }
I20260925 02:57:02.911911 2617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:02.913968 2619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:02.914271 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:02.906708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34515 (local address 127.12.222.254:36457)
0925 02:57:02.906800 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:02.906804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:02.906817 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:57:02.906854 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:02.906856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:02.906893 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:02.906976 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:02.907520 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:02.908282 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:02.909178 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:02.909481 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:02.909582 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:02.911096 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:02.911099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:02.911120 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:02.911148 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:02.912695 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:02.913111 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:02.913116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:02.913118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:02.913192 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:02.913500 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:02.913505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:02.913507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:02.913679 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:02.913800 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:02.914062 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 02:57:02.914139 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:57:02.919457 2619 catalog_manager.cc:1347] Generated new cluster ID: c6c27773437947a5ad3fb55c1b1a011b
I20260925 02:57:02.919514 2619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:02.941609 2619 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:02.942412 2619 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:02.951679 2619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 372626cabea24be6aa91ee9374bd77ec: Generated new TSK 0
I20260925 02:57:02.952293 2619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:57:02.962286 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 2532
W20260925 02:57:02.967319 2517 connection.cc:535] client connection to 127.12.222.254:36457 recv error: Network error: failed to read from TLS socket (remote: 127.12.222.254:36457): Connection reset by peer (error 104)
2026-09-25T02:57:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:57:03 dist-test-slave-252r krb5kdc[2631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:57:03 dist-test-slave-252r krb5kdc[2631](info): set up 2 sockets
Sep 25 02:57:03 dist-test-slave-252r krb5kdc[2631](info): commencing operation
krb5kdc: starting...
W20260925 02:57:03.940128 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:57:04 dist-test-slave-252r krb5kdc[2631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:57:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:57:04.047080 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:41145
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:41145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:04.148304 2644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:04.148537 2644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:04.148605 2644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:04.152441 2644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:04.152510 2644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:04.152546 2644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:04.152602 2644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:04.152640 2644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:04.157390 2644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:41145
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:41145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:57:04.158258 2644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:04.159117 2644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:04.166838 2650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:04.166838 2649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:04.166994 2652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:04.167270 2644 server_base.cc:1034] running on GCE node
I20260925 02:57:04.167827 2644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:04.168929 2644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:04.170109 2644 hybrid_clock.cc:648] HybridClock initialized: now 1790305024170099 us; error 24 us; skew 500 ppm
Sep 25 02:57:04 dist-test-slave-252r krb5kdc[2631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305024, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:04.172761 2644 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:57:04.173583 2644 webserver.cc:460] Webserver started at http://127.12.222.254:35093/ using document root <none> and password file <none>
I20260925 02:57:04.174098 2644 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:04.174185 2644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:04.174398 2644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:04.176359 2644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "0e2833086c3d4fb39cc5e74a5f22e01d"
format_stamp: "Formatted at 2026-09-25 02:57:04 on dist-test-slave-252r"
server_key: "64e253ab294c749ed2c78fc13f2e8b76"
server_key_iv: "353e0dc4dd8fddf9448601d32c71a652"
server_key_version: "encryptionkey@0"
I20260925 02:57:04.176954 2644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "0e2833086c3d4fb39cc5e74a5f22e01d"
format_stamp: "Formatted at 2026-09-25 02:57:04 on dist-test-slave-252r"
server_key: "64e253ab294c749ed2c78fc13f2e8b76"
server_key_iv: "353e0dc4dd8fddf9448601d32c71a652"
server_key_version: "encryptionkey@0"
I20260925 02:57:04.180673 2644 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:57:04.183051 2659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:04.184346 2644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:57:04.184475 2644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "0e2833086c3d4fb39cc5e74a5f22e01d"
format_stamp: "Formatted at 2026-09-25 02:57:04 on dist-test-slave-252r"
server_key: "64e253ab294c749ed2c78fc13f2e8b76"
server_key_iv: "353e0dc4dd8fddf9448601d32c71a652"
server_key_version: "encryptionkey@0"
I20260925 02:57:04.184579 2644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:04.198871 2644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:04.201764 2644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:04.201959 2644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:04.208014 2644 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:41145
I20260925 02:57:04.208027 2711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:41145 every 8 connection(s)
I20260925 02:57:04.208881 2644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:04.211200 2712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:04.211481 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 2644
I20260925 02:57:04.211617 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:57:04.211877 13179 external_mini_cluster.cc:1428] Setting key 4ec8798103665eb4f8eda5eb1504a15c
I20260925 02:57:04.217521 2712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d: Bootstrap starting.
I20260925 02:57:04.219938 2712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:04.220728 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:04.213033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39369 (local address 127.12.222.254:41145)
0925 02:57:04.213382 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:04.213398 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:04.213435 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:57:04.214106 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:04.214129 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:04.214504 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:04.214836 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:04.215182 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:04.216654 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:04.217462 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:04.218050 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:04.218191 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:04.218917 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:04.218922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:04.218935 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:04.219386 (+ 451us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:04.219454 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:04.219480 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:04.219617 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:04.219773 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 02:57:04.219903 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":82,"threads_started":1}
I20260925 02:57:04.220713 2712 log.cc:824] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:04.222568 2712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d: No bootstrap required, opened a new log
I20260925 02:57:04.223205 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:57:04.225919 2712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2833086c3d4fb39cc5e74a5f22e01d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 41145 } }
I20260925 02:57:04.226106 2712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:04.226181 2712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2833086c3d4fb39cc5e74a5f22e01d, State: Initialized, Role: FOLLOWER
I20260925 02:57:04.226611 2712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2833086c3d4fb39cc5e74a5f22e01d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 41145 } }
I20260925 02:57:04.226755 2712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:04.226837 2712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:04.226936 2712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:04.227669 2712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2833086c3d4fb39cc5e74a5f22e01d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 41145 } }
I20260925 02:57:04.227967 2712 leader_election.cc:302] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e2833086c3d4fb39cc5e74a5f22e01d; no voters:
I20260925 02:57:04.228259 2712 leader_election.cc:288] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:04.228327 2722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:04.228487 2722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d [term 1 LEADER]: Becoming Leader. State: Replica: 0e2833086c3d4fb39cc5e74a5f22e01d, State: Running, Role: LEADER
I20260925 02:57:04.228792 2722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2833086c3d4fb39cc5e74a5f22e01d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 41145 } }
I20260925 02:57:04.229354 2712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:04.230060 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:04.225022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35883 (local address 127.12.222.254:41145)
0925 02:57:04.225119 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:04.225122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:04.226372 (+ 1250us) server_negotiation.cc:407] Connection header received
0925 02:57:04.226481 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:04.226484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:04.226525 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:04.226602 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:04.226870 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:04.227367 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:04.228185 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:04.228542 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:04.228633 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:04.229445 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:04.229448 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:04.229451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:04.229461 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:04.229483 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:04.229499 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:04.229623 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:04.229889 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 02:57:04.229921 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:57:04.230432 2723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e2833086c3d4fb39cc5e74a5f22e01d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2833086c3d4fb39cc5e74a5f22e01d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 41145 } } }
I20260925 02:57:04.230495 2724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e2833086c3d4fb39cc5e74a5f22e01d. Latest consensus state: current_term: 1 leader_uuid: "0e2833086c3d4fb39cc5e74a5f22e01d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2833086c3d4fb39cc5e74a5f22e01d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 41145 } } }
I20260925 02:57:04.230558 2723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:04.230558 2724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:04.231040 2726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:04.233876 2726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:04.239745 2726 catalog_manager.cc:1347] Generated new cluster ID: 93b0c9ece2f94ebdb8885336c1f4d315
I20260925 02:57:04.239838 2726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:04.261428 2726 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:04.262282 2726 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:04.274412 2726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e2833086c3d4fb39cc5e74a5f22e01d: Generated new TSK 0
I20260925 02:57:04.275077 2726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:57:04.282624 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 2644
W20260925 02:57:04.288447 2629 connection.cc:535] client connection to 127.12.222.254:41145 recv error: Network error: failed to read from TLS socket (remote: 127.12.222.254:41145): Connection reset by peer (error 104)
2026-09-25T02:57:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:57:04 dist-test-slave-252r krb5kdc[2743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:57:04 dist-test-slave-252r krb5kdc[2743](info): set up 2 sockets
Sep 25 02:57:04 dist-test-slave-252r krb5kdc[2743](info): commencing operation
krb5kdc: starting...
W20260925 02:57:05.252118 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:57:05 dist-test-slave-252r krb5kdc[2743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:57:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:57:05.360664 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:33221
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:33221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:05.460108 2756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:05.460366 2756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:05.460451 2756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:05.464082 2756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:05.464161 2756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:05.464196 2756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:05.464246 2756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:05.464278 2756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:05.469043 2756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:33221
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:33221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:57:05.469937 2756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:05.470866 2756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:05.478713 2761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:05.478822 2756 server_base.cc:1034] running on GCE node
W20260925 02:57:05.478700 2764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:05.478749 2762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:05.479504 2756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:05.480545 2756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:05.481724 2756 hybrid_clock.cc:648] HybridClock initialized: now 1790305025481705 us; error 29 us; skew 500 ppm
Sep 25 02:57:05 dist-test-slave-252r krb5kdc[2743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305025, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:05.484170 2756 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:57:05.484987 2756 webserver.cc:460] Webserver started at http://127.12.222.254:46825/ using document root <none> and password file <none>
I20260925 02:57:05.485661 2756 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:05.485764 2756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:05.486001 2756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:05.488517 2756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "983001223308403d8d6a6af4d623a164"
format_stamp: "Formatted at 2026-09-25 02:57:05 on dist-test-slave-252r"
server_key: "82b716b8b4a25bae2d29560f2a9547c8"
server_key_iv: "915c78d223ca5de5aacde5c2b8cc26f7"
server_key_version: "encryptionkey@0"
I20260925 02:57:05.489115 2756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "983001223308403d8d6a6af4d623a164"
format_stamp: "Formatted at 2026-09-25 02:57:05 on dist-test-slave-252r"
server_key: "82b716b8b4a25bae2d29560f2a9547c8"
server_key_iv: "915c78d223ca5de5aacde5c2b8cc26f7"
server_key_version: "encryptionkey@0"
I20260925 02:57:05.492357 2756 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:57:05.494539 2771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:05.495728 2756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:57:05.495860 2756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "983001223308403d8d6a6af4d623a164"
format_stamp: "Formatted at 2026-09-25 02:57:05 on dist-test-slave-252r"
server_key: "82b716b8b4a25bae2d29560f2a9547c8"
server_key_iv: "915c78d223ca5de5aacde5c2b8cc26f7"
server_key_version: "encryptionkey@0"
I20260925 02:57:05.495971 2756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:05.500078 2756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:05.503263 2756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:05.503437 2756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:05.510087 2756 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:33221
I20260925 02:57:05.510113 2823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:33221 every 8 connection(s)
I20260925 02:57:05.510898 2756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:05.513273 2824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:05.515429 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 2756
I20260925 02:57:05.515587 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:57:05.515893 13179 external_mini_cluster.cc:1428] Setting key a89d3c929e88718407037c2500bf6de2
I20260925 02:57:05.518920 2824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164: Bootstrap starting.
I20260925 02:57:05.520787 2827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:05.517239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50137 (local address 127.12.222.254:33221)
0925 02:57:05.517650 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:05.517663 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:05.517693 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:57:05.518277 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:05.518292 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:05.518543 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:05.518848 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:05.519632 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:05.519637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:05.519657 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:05.519666 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:05.519715 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:05.519744 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:05.519875 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:05.519967 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 02:57:05.520100 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":135,"threads_started":1}
I20260925 02:57:05.521198 2824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:05.522010 2824 log.cc:824] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:05.523376 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:57:05.523940 2824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164: No bootstrap required, opened a new log
I20260925 02:57:05.526736 2827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:05.525369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32941 (local address 127.12.222.254:33221)
0925 02:57:05.525473 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:05.525478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:05.525636 (+ 158us) server_negotiation.cc:407] Connection header received
0925 02:57:05.525744 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:05.525746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:05.525771 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:05.525831 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:05.526401 (+ 570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:05.526403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:05.526405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:05.526414 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:05.526435 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:05.526444 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:05.526506 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:05.526605 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 02:57:05.526630 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:57:05.528012 2824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983001223308403d8d6a6af4d623a164" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33221 } }
I20260925 02:57:05.528225 2824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:05.528318 2824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983001223308403d8d6a6af4d623a164, State: Initialized, Role: FOLLOWER
I20260925 02:57:05.528671 2824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983001223308403d8d6a6af4d623a164" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33221 } }
I20260925 02:57:05.528810 2824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:05.528882 2824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:05.528973 2824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:05.529768 2824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983001223308403d8d6a6af4d623a164" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33221 } }
I20260925 02:57:05.530094 2824 leader_election.cc:302] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 983001223308403d8d6a6af4d623a164; no voters:
I20260925 02:57:05.530441 2824 leader_election.cc:288] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:05.530500 2834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:05.530699 2834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164 [term 1 LEADER]: Becoming Leader. State: Replica: 983001223308403d8d6a6af4d623a164, State: Running, Role: LEADER
I20260925 02:57:05.531028 2834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983001223308403d8d6a6af4d623a164" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33221 } }
I20260925 02:57:05.531586 2824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:05.532969 2836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "983001223308403d8d6a6af4d623a164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983001223308403d8d6a6af4d623a164" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33221 } } }
I20260925 02:57:05.532970 2835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 983001223308403d8d6a6af4d623a164. Latest consensus state: current_term: 1 leader_uuid: "983001223308403d8d6a6af4d623a164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983001223308403d8d6a6af4d623a164" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33221 } } }
I20260925 02:57:05.533150 2836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:05.533159 2835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164 [sys.catalog]: This master's current role is: LEADER
W20260925 02:57:05.536118 2850 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:57:05.536221 2850 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:57:05.536329 2843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:05.536998 2843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:05.541611 2843 catalog_manager.cc:1347] Generated new cluster ID: fa0f0d74537449f196bef1ef26c597bc
I20260925 02:57:05.541669 2843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:05.568826 2843 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:05.569075 2843 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:05.585729 2843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 983001223308403d8d6a6af4d623a164: Generated new TSK 0
I20260925 02:57:05.586325 2843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:57:05.591339 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 2756
2026-09-25T02:57:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:57:05 dist-test-slave-252r krb5kdc[2855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:57:05 dist-test-slave-252r krb5kdc[2855](info): set up 2 sockets
Sep 25 02:57:05 dist-test-slave-252r krb5kdc[2855](info): commencing operation
krb5kdc: starting...
W20260925 02:57:06.552278 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:57:06 dist-test-slave-252r krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:57:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:57:06.656769 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:43367
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:43367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:06.758605 2868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:06.758836 2868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:06.758905 2868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:06.762769 2868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:06.762846 2868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:06.762897 2868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:06.762931 2868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:06.762969 2868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:06.767784 2868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:43367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:43367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:57:06.768720 2868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:06.769583 2868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:06.777041 2874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:06.777154 2868 server_base.cc:1034] running on GCE node
W20260925 02:57:06.777041 2873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:06.777068 2876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:06.777844 2868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:06.778867 2868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:06.780055 2868 hybrid_clock.cc:648] HybridClock initialized: now 1790305026780039 us; error 27 us; skew 500 ppm
Sep 25 02:57:06 dist-test-slave-252r krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305026, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:06.788820 2868 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:57:06.789608 2868 webserver.cc:460] Webserver started at http://127.12.222.254:37213/ using document root <none> and password file <none>
I20260925 02:57:06.790055 2868 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:06.790153 2868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:06.790308 2868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:06.792133 2868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "ca1e14079cf642e8aaf0e6f991e17e11"
format_stamp: "Formatted at 2026-09-25 02:57:06 on dist-test-slave-252r"
server_key: "b2e71d05f6f002c3be974f0fa22c78df"
server_key_iv: "76aa88f9f1ac06a7a53b9d93b0384313"
server_key_version: "encryptionkey@0"
I20260925 02:57:06.792735 2868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "ca1e14079cf642e8aaf0e6f991e17e11"
format_stamp: "Formatted at 2026-09-25 02:57:06 on dist-test-slave-252r"
server_key: "b2e71d05f6f002c3be974f0fa22c78df"
server_key_iv: "76aa88f9f1ac06a7a53b9d93b0384313"
server_key_version: "encryptionkey@0"
I20260925 02:57:06.796101 2868 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:57:06.798466 2883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:06.799814 2868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:57:06.799943 2868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "ca1e14079cf642e8aaf0e6f991e17e11"
format_stamp: "Formatted at 2026-09-25 02:57:06 on dist-test-slave-252r"
server_key: "b2e71d05f6f002c3be974f0fa22c78df"
server_key_iv: "76aa88f9f1ac06a7a53b9d93b0384313"
server_key_version: "encryptionkey@0"
I20260925 02:57:06.800040 2868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:06.813941 2868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:06.816529 2868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:06.816708 2868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:06.823074 2868 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:43367
I20260925 02:57:06.823144 2935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:43367 every 8 connection(s)
I20260925 02:57:06.823926 2868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:06.826460 2936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:06.831403 2936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11: Bootstrap starting.
I20260925 02:57:06.831597 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 2868
I20260925 02:57:06.831732 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:57:06.831970 13179 external_mini_cluster.cc:1428] Setting key 98cd372fdcda28e994bd6525880652f5
I20260925 02:57:06.833864 2936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:06.834683 2936 log.cc:824] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:06.836812 2936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11: No bootstrap required, opened a new log
I20260925 02:57:06.840274 2936 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1e14079cf642e8aaf0e6f991e17e11" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43367 } }
I20260925 02:57:06.840493 2936 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:06.840570 2936 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1e14079cf642e8aaf0e6f991e17e11, State: Initialized, Role: FOLLOWER
Sep 25 02:57:06 dist-test-slave-252r krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:57:06.841040 2936 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1e14079cf642e8aaf0e6f991e17e11" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43367 } }
I20260925 02:57:06.841202 2936 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:06.841274 2936 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:06.841365 2936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:06.842108 2936 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1e14079cf642e8aaf0e6f991e17e11" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43367 } }
I20260925 02:57:06.842401 2936 leader_election.cc:302] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca1e14079cf642e8aaf0e6f991e17e11; no voters:
I20260925 02:57:06.842696 2936 leader_election.cc:288] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:06.842756 2941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:06.842981 2941 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11 [term 1 LEADER]: Becoming Leader. State: Replica: ca1e14079cf642e8aaf0e6f991e17e11, State: Running, Role: LEADER
I20260925 02:57:06.843286 2941 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1e14079cf642e8aaf0e6f991e17e11" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43367 } }
I20260925 02:57:06.843883 2936 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:06.844724 2942 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca1e14079cf642e8aaf0e6f991e17e11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1e14079cf642e8aaf0e6f991e17e11" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43367 } } }
I20260925 02:57:06.844849 2942 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:06.845052 2942 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca1e14079cf642e8aaf0e6f991e17e11. Latest consensus state: current_term: 1 leader_uuid: "ca1e14079cf642e8aaf0e6f991e17e11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1e14079cf642e8aaf0e6f991e17e11" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43367 } } }
I20260925 02:57:06.845120 2942 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:06.845851 2950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:06.846345 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.833148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44943 (local address 127.12.222.254:43367)
0925 02:57:06.833548 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:06.833563 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:06.833599 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:57:06.834347 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:06.834369 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:06.834778 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:06.835125 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:06.836026 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.837378 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.838152 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.838780 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.838930 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.841299 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:06.841306 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:06.841322 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:06.841369 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:06.843075 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:06.843541 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:06.843545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:06.843548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:06.843642 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:06.843921 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:06.843924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:06.843925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:06.844395 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:06.845076 (+ 681us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:06.845205 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 02:57:06.845463 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":98,"threads_started":1}
I20260925 02:57:06.848749 2950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:06.849048 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:57:06.854086 2950 catalog_manager.cc:1347] Generated new cluster ID: ef4e088fe9e546e5a8bfea4e3e4b4270
I20260925 02:57:06.854138 2950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:06.859589 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:06.851080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39121 (local address 127.12.222.254:43367)
0925 02:57:06.851169 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:06.851172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:06.851400 (+ 228us) server_negotiation.cc:407] Connection header received
0925 02:57:06.851486 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:06.851488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:06.851528 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:06.851605 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:06.852280 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.852861 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.853953 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:06.854300 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:06.854405 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:06.856006 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:06.856010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:06.856013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:06.856049 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:06.857595 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:06.858062 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:06.858066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:06.858067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:06.858141 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:06.858431 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:06.858434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:06.858436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:06.858617 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:06.858738 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:06.859367 (+ 629us) server_negotiation.cc:299] Negotiation successful
0925 02:57:06.859462 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:57:06.865021 2950 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:06.866030 2950 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:06.876657 2950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca1e14079cf642e8aaf0e6f991e17e11: Generated new TSK 0
I20260925 02:57:06.877394 2950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:57:06.885120 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 2868
2026-09-25T02:57:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:57:06 dist-test-slave-252r krb5kdc[2967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:57:06 dist-test-slave-252r krb5kdc[2967](info): set up 2 sockets
Sep 25 02:57:06 dist-test-slave-252r krb5kdc[2967](info): commencing operation
krb5kdc: starting...
W20260925 02:57:07.856653 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:57:07 dist-test-slave-252r krb5kdc[2967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:57:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:57:07.962324 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:43727
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:43727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:08.063513 2980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:08.063737 2980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:08.063773 2980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:08.067533 2980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:08.067576 2980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:08.067592 2980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:08.067624 2980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:08.067638 2980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:08.072368 2980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:43727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:43727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:57:08.073179 2980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:08.074007 2980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:08.081657 2988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:08.081640 2985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:08.081643 2986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:08.081760 2980 server_base.cc:1034] running on GCE node
I20260925 02:57:08.082477 2980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:08.083420 2980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:08.084580 2980 hybrid_clock.cc:648] HybridClock initialized: now 1790305028084546 us; error 33 us; skew 500 ppm
Sep 25 02:57:08 dist-test-slave-252r krb5kdc[2967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305028, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:08.087195 2980 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:57:08.088006 2980 webserver.cc:460] Webserver started at http://127.12.222.254:42423/ using document root <none> and password file <none>
I20260925 02:57:08.088567 2980 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:08.088653 2980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:08.088867 2980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:08.090667 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "0fe62e51101f437199c6af31ec9e6776"
format_stamp: "Formatted at 2026-09-25 02:57:08 on dist-test-slave-252r"
server_key: "a5b09305da27e2dd55513cf84eff80b4"
server_key_iv: "60c4f908b90f4be7122b9740c24887f6"
server_key_version: "encryptionkey@0"
I20260925 02:57:08.091176 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "0fe62e51101f437199c6af31ec9e6776"
format_stamp: "Formatted at 2026-09-25 02:57:08 on dist-test-slave-252r"
server_key: "a5b09305da27e2dd55513cf84eff80b4"
server_key_iv: "60c4f908b90f4be7122b9740c24887f6"
server_key_version: "encryptionkey@0"
I20260925 02:57:08.094426 2980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:57:08.096555 2995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:08.097770 2980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 02:57:08.097899 2980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "0fe62e51101f437199c6af31ec9e6776"
format_stamp: "Formatted at 2026-09-25 02:57:08 on dist-test-slave-252r"
server_key: "a5b09305da27e2dd55513cf84eff80b4"
server_key_iv: "60c4f908b90f4be7122b9740c24887f6"
server_key_version: "encryptionkey@0"
I20260925 02:57:08.097990 2980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:08.115936 2980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:08.118372 2980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:08.118546 2980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:08.124744 2980 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:43727
I20260925 02:57:08.124810 3047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:43727 every 8 connection(s)
I20260925 02:57:08.125608 2980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:08.127329 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 2980
I20260925 02:57:08.127455 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:57:08.127719 13179 external_mini_cluster.cc:1428] Setting key 8f9ab92ff00dc8f77f7b16d264d5aa9e
I20260925 02:57:08.128324 3048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:08.134449 3048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776: Bootstrap starting.
I20260925 02:57:08.136552 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:08.128835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34993 (local address 127.12.222.254:43727)
0925 02:57:08.129224 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:08.129241 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:08.129294 (+ 53us) server_negotiation.cc:407] Connection header received
0925 02:57:08.130059 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:08.130081 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:08.130473 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:08.130794 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:08.131147 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:08.132496 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:08.133310 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:08.133951 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:08.134100 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:08.134788 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:08.134794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:08.134806 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:08.135200 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:08.135259 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:08.135286 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:08.135376 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:08.135498 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 02:57:08.135691 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":111,"threads_started":1}
I20260925 02:57:08.136979 3048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:08.137753 3048 log.cc:824] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:08.139168 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:57:08.139726 3048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776: No bootstrap required, opened a new log
I20260925 02:57:08.143900 3048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe62e51101f437199c6af31ec9e6776" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43727 } }
I20260925 02:57:08.144150 3048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:08.144214 3048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe62e51101f437199c6af31ec9e6776, State: Initialized, Role: FOLLOWER
I20260925 02:57:08.144642 3048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe62e51101f437199c6af31ec9e6776" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43727 } }
I20260925 02:57:08.144783 3048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:08.144847 3048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:08.144944 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:08.141246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46893 (local address 127.12.222.254:43727)
0925 02:57:08.141362 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:08.141366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:08.141528 (+ 162us) server_negotiation.cc:407] Connection header received
0925 02:57:08.141610 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:08.141614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:08.141656 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:08.141729 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:08.141957 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:08.142714 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:08.143430 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:08.143723 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:08.143834 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:08.144511 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:08.144514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:08.144517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:08.144532 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:08.144552 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:08.144561 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:08.144632 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:08.144744 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 02:57:08.144786 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:57:08.144950 3048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:08.145992 3048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe62e51101f437199c6af31ec9e6776" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43727 } }
I20260925 02:57:08.146311 3048 leader_election.cc:302] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fe62e51101f437199c6af31ec9e6776; no voters:
I20260925 02:57:08.146589 3048 leader_election.cc:288] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:08.146675 3058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:08.146858 3058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776 [term 1 LEADER]: Becoming Leader. State: Replica: 0fe62e51101f437199c6af31ec9e6776, State: Running, Role: LEADER
I20260925 02:57:08.147202 3058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe62e51101f437199c6af31ec9e6776" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43727 } }
I20260925 02:57:08.147692 3048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:08.148732 3059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fe62e51101f437199c6af31ec9e6776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe62e51101f437199c6af31ec9e6776" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43727 } } }
I20260925 02:57:08.148870 3059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:08.148993 3060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fe62e51101f437199c6af31ec9e6776. Latest consensus state: current_term: 1 leader_uuid: "0fe62e51101f437199c6af31ec9e6776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe62e51101f437199c6af31ec9e6776" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 43727 } } }
I20260925 02:57:08.149148 3060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:08.149763 3068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:08.152395 3068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:08.157604 3068 catalog_manager.cc:1347] Generated new cluster ID: 98911251d4624dd5a0a52df879263b9d
I20260925 02:57:08.157671 3068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:08.169029 3068 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:08.170109 3068 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:08.188514 3068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fe62e51101f437199c6af31ec9e6776: Generated new TSK 0
I20260925 02:57:08.189157 3068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:57:08.194406 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 2980
2026-09-25T02:57:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:57:08 dist-test-slave-252r krb5kdc[3079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:57:08 dist-test-slave-252r krb5kdc[3079](info): set up 2 sockets
Sep 25 02:57:08 dist-test-slave-252r krb5kdc[3079](info): commencing operation
krb5kdc: starting...
W20260925 02:57:09.176285 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:57:09 dist-test-slave-252r krb5kdc[3079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:57:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:57:09.283398 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:44865
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:44865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:09.384052 3092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:09.384305 3092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:09.384346 3092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:09.388100 3092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:09.388144 3092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:09.388164 3092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:09.388180 3092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:09.388195 3092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:09.392939 3092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:44865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:44865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:57:09.393666 3092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:09.394485 3092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:09.401988 3097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:09.402082 3098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:09.401988 3100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:09.402244 3092 server_base.cc:1034] running on GCE node
I20260925 02:57:09.402776 3092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:09.403723 3092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:09.404906 3092 hybrid_clock.cc:648] HybridClock initialized: now 1790305029404891 us; error 30 us; skew 500 ppm
Sep 25 02:57:09 dist-test-slave-252r krb5kdc[3079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305029, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:09.407382 3092 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:57:09.408181 3092 webserver.cc:460] Webserver started at http://127.12.222.254:35827/ using document root <none> and password file <none>
I20260925 02:57:09.408715 3092 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:09.408774 3092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:09.409011 3092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:09.410780 3092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "428424338428448c8db9937d3e26ad17"
format_stamp: "Formatted at 2026-09-25 02:57:09 on dist-test-slave-252r"
server_key: "9ae589427e603bfe300a073a059fe378"
server_key_iv: "62d355f1109e33a7d5f03884c39a3448"
server_key_version: "encryptionkey@0"
I20260925 02:57:09.411319 3092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "428424338428448c8db9937d3e26ad17"
format_stamp: "Formatted at 2026-09-25 02:57:09 on dist-test-slave-252r"
server_key: "9ae589427e603bfe300a073a059fe378"
server_key_iv: "62d355f1109e33a7d5f03884c39a3448"
server_key_version: "encryptionkey@0"
I20260925 02:57:09.414628 3092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 02:57:09.416762 3107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:09.417815 3092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:57:09.417929 3092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "428424338428448c8db9937d3e26ad17"
format_stamp: "Formatted at 2026-09-25 02:57:09 on dist-test-slave-252r"
server_key: "9ae589427e603bfe300a073a059fe378"
server_key_iv: "62d355f1109e33a7d5f03884c39a3448"
server_key_version: "encryptionkey@0"
I20260925 02:57:09.418017 3092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:09.422210 3092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:09.424750 3092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:09.424945 3092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:09.431346 3092 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:44865
I20260925 02:57:09.431358 3159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:44865 every 8 connection(s)
I20260925 02:57:09.432165 3092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:09.434501 3160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:09.438073 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 3092
I20260925 02:57:09.438185 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:57:09.438431 13179 external_mini_cluster.cc:1428] Setting key b0cfa368544a11d41a202d102fb5c952
I20260925 02:57:09.439703 3160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17: Bootstrap starting.
I20260925 02:57:09.442185 3160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:09.443038 3160 log.cc:824] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:09.443544 3163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.439706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41721 (local address 127.12.222.254:44865)
0925 02:57:09.440115 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:09.440132 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:09.440168 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:57:09.441001 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:09.441023 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:09.441332 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:09.441667 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:09.442330 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:09.442335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:09.442349 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:09.442361 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:09.442433 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:09.442463 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:09.442573 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:09.442699 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 02:57:09.442823 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":141,"threads_started":1}
I20260925 02:57:09.444924 3160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17: No bootstrap required, opened a new log
I20260925 02:57:09.446089 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:57:09.448480 3160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428424338428448c8db9937d3e26ad17" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 44865 } }
I20260925 02:57:09.448693 3160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:57:09.448748 3160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428424338428448c8db9937d3e26ad17, State: Initialized, Role: FOLLOWER
I20260925 02:57:09.449151 3160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428424338428448c8db9937d3e26ad17" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 44865 } }
I20260925 02:57:09.449283 3160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:57:09.449347 3160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:57:09.449453 3160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:09.449484 3163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:09.447926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40465 (local address 127.12.222.254:44865)
0925 02:57:09.448033 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:09.448037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:09.448215 (+ 178us) server_negotiation.cc:407] Connection header received
0925 02:57:09.448353 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:09.448356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:09.448388 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:09.448459 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:09.449123 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:09.449126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:09.449128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:57:09.449138 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:09.449161 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:57:09.449171 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:09.449268 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:09.449335 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 02:57:09.449369 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:57:09.450529 3160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428424338428448c8db9937d3e26ad17" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 44865 } }
I20260925 02:57:09.450848 3160 leader_election.cc:302] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 428424338428448c8db9937d3e26ad17; no voters:
I20260925 02:57:09.451123 3160 leader_election.cc:288] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:57:09.451264 3170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:09.451504 3170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17 [term 1 LEADER]: Becoming Leader. State: Replica: 428424338428448c8db9937d3e26ad17, State: Running, Role: LEADER
I20260925 02:57:09.451828 3170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428424338428448c8db9937d3e26ad17" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 44865 } }
I20260925 02:57:09.452294 3160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:09.453343 3171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "428424338428448c8db9937d3e26ad17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428424338428448c8db9937d3e26ad17" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 44865 } } }
I20260925 02:57:09.453472 3171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:09.453815 3172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 428424338428448c8db9937d3e26ad17. Latest consensus state: current_term: 1 leader_uuid: "428424338428448c8db9937d3e26ad17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428424338428448c8db9937d3e26ad17" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 44865 } } }
I20260925 02:57:09.453963 3172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17 [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:09.453883 3179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:09.457226 3179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:09.461464 3179 catalog_manager.cc:1347] Generated new cluster ID: 9d68fd596605476e8553104523023a98
I20260925 02:57:09.461524 3179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:09.469786 3179 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:09.470042 3179 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:09.484915 3179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 428424338428448c8db9937d3e26ad17: Generated new TSK 0
I20260925 02:57:09.485496 3179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:57:09.490216 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 3092
2026-09-25T02:57:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1294 ms)
[----------] 18 tests from AuthTokenIssuingTest (23573 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:57:09 dist-test-slave-252r krb5kdc[3191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:57:09 dist-test-slave-252r krb5kdc[3191](info): set up 2 sockets
Sep 25 02:57:09 dist-test-slave-252r krb5kdc[3191](info): commencing operation
krb5kdc: starting...
W20260925 02:57:10.456171 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:57:10 dist-test-slave-252r krb5kdc[3191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:57:10.566632 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:38519
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:38519,127.12.222.253:44223,127.12.222.252:42659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:10.667090 3204 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:57:10.667320 3204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:10.667372 3204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:10.667440 3204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:10.671089 3204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:10.671131 3204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:10.671149 3204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:10.671196 3204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:10.671226 3204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:10.676574 3204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:38519,127.12.222.253:44223,127.12.222.252:42659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:38519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:57:10.677511 3204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:10.678361 3204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:10.685974 3210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:10.685957 3212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:10.685966 3209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:10.685999 3204 server_base.cc:1034] running on GCE node
I20260925 02:57:10.686738 3204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:10.687664 3204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:10.688840 3204 hybrid_clock.cc:648] HybridClock initialized: now 1790305030688823 us; error 28 us; skew 500 ppm
Sep 25 02:57:10 dist-test-slave-252r krb5kdc[3191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305030, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:10.691445 3204 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:57:10.692200 3204 webserver.cc:460] Webserver started at http://127.12.222.254:44269/ using document root <none> and password file <none>
I20260925 02:57:10.692689 3204 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:10.692772 3204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:10.692973 3204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:10.694710 3204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "9ec9563c17704f6987a2a1e09a12368d"
format_stamp: "Formatted at 2026-09-25 02:57:10 on dist-test-slave-252r"
server_key: "0156b63c194757f49755f7a2c33f5acb"
server_key_iv: "ab4e6af507ab54c5835041a87c20b632"
server_key_version: "encryptionkey@0"
I20260925 02:57:10.695250 3204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "9ec9563c17704f6987a2a1e09a12368d"
format_stamp: "Formatted at 2026-09-25 02:57:10 on dist-test-slave-252r"
server_key: "0156b63c194757f49755f7a2c33f5acb"
server_key_iv: "ab4e6af507ab54c5835041a87c20b632"
server_key_version: "encryptionkey@0"
I20260925 02:57:10.698604 3204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 02:57:10.700722 3219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:10.701808 3204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:57:10.701941 3204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "9ec9563c17704f6987a2a1e09a12368d"
format_stamp: "Formatted at 2026-09-25 02:57:10 on dist-test-slave-252r"
server_key: "0156b63c194757f49755f7a2c33f5acb"
server_key_iv: "ab4e6af507ab54c5835041a87c20b632"
server_key_version: "encryptionkey@0"
I20260925 02:57:10.702042 3204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:10.718300 3204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:10.721050 3204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:10.721226 3204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:10.727126 3204 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:38519
I20260925 02:57:10.727416 3271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:38519 every 8 connection(s)
I20260925 02:57:10.727952 3204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:10.730273 3272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:10.731588 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 3204
I20260925 02:57:10.731702 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:57:10.731997 13179 external_mini_cluster.cc:1428] Setting key 2b7c9c16336d7ddebd7fdd88e91570e1
I20260925 02:57:10.734495 3272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.222.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.253@KRBTEST.COM" created.
Sep 25 02:57:10 dist-test-slave-252r krb5kdc[3191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305030, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:57:10.748207 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.736448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.12.222.254:38519)
0925 02:57:10.736811 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:10.736828 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:10.737080 (+ 252us) server_negotiation.cc:407] Connection header received
0925 02:57:10.738021 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:10.738025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:10.738177 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:10.738346 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:10.739098 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.739947 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.740930 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.741277 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.741439 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:10.743906 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:10.743912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:10.743920 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:10.743939 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:10.745461 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:10.745918 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:10.745923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:10.745925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:10.746027 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:10.746330 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:10.746336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:10.746338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:10.746646 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:10.746786 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:10.747127 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 02:57:10.747351 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":237,"thread_start_us":119,"threads_started":1}
I20260925 02:57:10.748207 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.736449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38519 (local address 127.0.0.1:34726)
0925 02:57:10.736952 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:10.736985 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:10.737742 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:10.738346 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:10.738351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:10.738753 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:10.738959 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:10.738971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.740077 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.740079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:10.740812 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:10.740822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.741427 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.741431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:10.741521 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:10.742091 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:10.742115 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:10.743757 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:10.745572 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:10.745582 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:10.745591 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:10.745806 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:10.746127 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:10.746130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:10.746132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:10.746240 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:10.746763 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:10.746768 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:10.746901 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 02:57:10.747056 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 02:57:10.747410 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":102,"thread_start_us":137,"threads_started":1}
I20260925 02:57:10.750134 3272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:57:10.750778 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.750537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:44223 (local address 127.0.0.1:51534)
0925 02:57:10.750640 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:10.750701 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 02:57:10.750974 3223 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44223: connect: Connection refused (error 111)
I20260925 02:57:10.751735 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.751556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:44223 (local address 127.0.0.1:51538)
0925 02:57:10.751648 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:10.751689 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 02:57:10.751912 3272 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.253:44223: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44223: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.253.keytab.
Entry for principal kudu/127.12.222.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.253.keytab.
I20260925 02:57:10.756414 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.253:44223
--webserver_interface=127.12.222.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.253.keytab
--principal=kudu/127.12.222.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:38519,127.12.222.253:44223,127.12.222.252:42659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:57:10.801151 3272 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } attempt: 1
I20260925 02:57:10.801977 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.801703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:44223 (local address 127.0.0.1:51542)
0925 02:57:10.801849 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:10.801909 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 02:57:10.802749 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.802564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:44223 (local address 127.0.0.1:51554)
0925 02:57:10.802658 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:10.802700 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 02:57:10.802939 3272 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.253:44223: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44223: connect: Connection refused (error 111)
W20260925 02:57:10.858144 3279 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:57:10.858397 3279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:10.858458 3279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:10.858529 3279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:10.862332 3279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:10.862425 3279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:10.862458 3279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:10.862534 3279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:10.862569 3279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:10.867441 3279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:38519,127.12.222.253:44223,127.12.222.252:42659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.253.keytab
--principal=kudu/127.12.222.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.253:44223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.222.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:57:10.868376 3279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:10.869285 3279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:57:10.871163 3272 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } attempt: 2
I20260925 02:57:10.871969 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.871674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:44223 (local address 127.0.0.1:51560)
0925 02:57:10.871844 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:10.871904 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 02:57:10.872722 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.872501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:44223 (local address 127.0.0.1:51568)
0925 02:57:10.872611 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:10.872665 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 02:57:10.872848 3272 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.253:44223: Network error: Client connection negotiation failed: client connection to 127.12.222.253:44223: connect: Connection refused (error 111)
W20260925 02:57:10.877193 3287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:10.877202 3285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:10.877202 3284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:10.877277 3279 server_base.cc:1034] running on GCE node
I20260925 02:57:10.877945 3279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:10.878881 3279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:10.880040 3279 hybrid_clock.cc:648] HybridClock initialized: now 1790305030880017 us; error 35 us; skew 500 ppm
Sep 25 02:57:10 dist-test-slave-252r krb5kdc[3191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305030, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:10.882604 3279 init.cc:377] Logged in from keytab as kudu/127.12.222.253@KRBTEST.COM (short username kudu)
I20260925 02:57:10.883332 3279 webserver.cc:460] Webserver started at http://127.12.222.253:33459/ using document root <none> and password file <none>
I20260925 02:57:10.883767 3279 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:10.883821 3279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:10.883971 3279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:10.885708 3279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-1/data/instance:
uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77"
format_stamp: "Formatted at 2026-09-25 02:57:10 on dist-test-slave-252r"
server_key: "874a9bd038b0f2ea1200717a3d81a45e"
server_key_iv: "3594cb2656820c30421aa03e28e03adf"
server_key_version: "encryptionkey@0"
I20260925 02:57:10.886202 3279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-1/wal/instance:
uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77"
format_stamp: "Formatted at 2026-09-25 02:57:10 on dist-test-slave-252r"
server_key: "874a9bd038b0f2ea1200717a3d81a45e"
server_key_iv: "3594cb2656820c30421aa03e28e03adf"
server_key_version: "encryptionkey@0"
I20260925 02:57:10.889384 3279 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:57:10.891397 3294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:10.892520 3279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:57:10.892634 3279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-1/wal
uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77"
format_stamp: "Formatted at 2026-09-25 02:57:10 on dist-test-slave-252r"
server_key: "874a9bd038b0f2ea1200717a3d81a45e"
server_key_iv: "3594cb2656820c30421aa03e28e03adf"
server_key_version: "encryptionkey@0"
I20260925 02:57:10.892725 3279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:10.906713 3279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:10.909601 3279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:10.909766 3279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:10.915642 3279 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.253:44223
I20260925 02:57:10.915896 3346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.253:44223 every 8 connection(s)
I20260925 02:57:10.916558 3279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-1/data/info.pb
I20260925 02:57:10.919031 3347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:10.921479 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 3279
I20260925 02:57:10.921635 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-1/wal/instance
I20260925 02:57:10.921962 13179 external_mini_cluster.cc:1428] Setting key ad60b1fa129ad8c0382a5b5017ab8e74
I20260925 02:57:10.922834 3347 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.222.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.252@KRBTEST.COM" created.
Sep 25 02:57:10 dist-test-slave-252r krb5kdc[3191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305030, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.253@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:57:10.937052 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.924534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34742 (local address 127.12.222.254:38519)
0925 02:57:10.924664 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:10.924670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:10.925487 (+ 817us) server_negotiation.cc:407] Connection header received
0925 02:57:10.926261 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:10.926267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:10.926332 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:10.926429 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:10.927399 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.927875 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.928981 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.929199 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.929303 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:10.933626 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:10.933629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:10.933630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:10.933653 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:10.935022 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:10.935505 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:10.935509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:10.935511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:10.935587 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:10.935925 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:10.935928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:10.935930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:10.936099 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:10.936224 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:10.936770 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 02:57:10.936895 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:57:10.937702 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.924916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38519 (local address 127.0.0.1:34742)
0925 02:57:10.925318 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:10.925369 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:10.926053 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:10.926562 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:10.926567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:10.926852 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:10.927246 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:10.927257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.928002 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.928005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:10.928858 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:10.928867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.929336 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.929340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:10.929497 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:10.930031 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:10.930046 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:10.933470 (+ 3424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:10.935139 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:10.935144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:10.935156 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:10.935402 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:10.935685 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:10.935688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:10.935690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:10.935824 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:10.936222 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:10.936253 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:10.936501 (+ 248us) client_negotiation.cc:769] Sending connection context
0925 02:57:10.936709 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 02:57:10.936979 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":104,"threads_started":1}
I20260925 02:57:10.938812 3347 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:57:10 dist-test-slave-252r krb5kdc[3191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305030, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.253@KRBTEST.COM for kudu/127.12.222.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.252.keytab.
Entry for principal kudu/127.12.222.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.252.keytab.
I20260925 02:57:10.945565 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.252:42659
--webserver_interface=127.12.222.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.252.keytab
--principal=kudu/127.12.222.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:38519,127.12.222.253:44223,127.12.222.252:42659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:57:10.947208 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.939246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:44223 (local address 127.0.0.1:51570)
0925 02:57:10.939352 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:10.939363 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:10.939427 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:10.939970 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:10.939971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:10.940079 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:10.940150 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:10.940154 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.941179 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.941182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:10.941691 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:10.941698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.942296 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.942298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:10.942358 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:10.942696 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:10.942706 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:10.943754 (+ 1048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:10.945700 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:10.945705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:10.945707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:10.945920 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:10.946246 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:10.946248 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:10.946249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:10.946318 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:10.946753 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:10.946756 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:10.946839 (+ 83us) client_negotiation.cc:769] Sending connection context
0925 02:57:10.946951 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:57:10.947060 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 02:57:10.947319 3352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.939355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51570 (local address 127.12.222.253:44223)
0925 02:57:10.939613 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:10.939622 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:10.939664 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:57:10.939720 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:10.939726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:10.939864 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:10.939964 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:10.940290 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.941075 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.941786 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.942205 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.942281 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:10.943873 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:10.943880 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:10.943894 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:10.943913 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:10.945569 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:10.946026 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:10.946034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:10.946043 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:10.946152 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:10.946426 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:10.946429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:10.946431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:10.946625 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:10.946770 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:10.947037 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 02:57:10.947144 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":119,"threads_started":1}
I20260925 02:57:10.948740 3347 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:57:10.949365 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.949149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:42659 (local address 127.0.0.1:47654)
0925 02:57:10.949250 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:10.949320 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 02:57:10.949541 3296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
I20260925 02:57:10.950250 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.950048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:42659 (local address 127.0.0.1:47656)
0925 02:57:10.950160 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:10.950206 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 02:57:10.950399 3347 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.252:42659: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
I20260925 02:57:10.964124 3272 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } attempt: 3
Sep 25 02:57:10 dist-test-slave-252r krb5kdc[3191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305030, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for kudu/127.12.222.253@KRBTEST.COM
I20260925 02:57:10.972213 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.964546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:44223 (local address 127.0.0.1:51578)
0925 02:57:10.964697 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:10.964710 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:10.964793 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:10.965016 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:10.965019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:10.965190 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:10.965276 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:10.965282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.965928 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.965932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:10.966716 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:10.966726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.967569 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.967572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:10.967678 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:10.968074 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:10.968086 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:10.969259 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:10.970823 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:10.970825 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:10.970827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:10.971029 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:10.971291 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:10.971294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:10.971296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:10.971377 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:10.971721 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:10.971724 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:10.971832 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 02:57:10.971940 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 02:57:10.972065 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 02:57:10.972213 3352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.964607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.12.222.253:44223)
0925 02:57:10.964704 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:10.964707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:10.964751 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:57:10.964869 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:10.964871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:10.964909 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:10.964974 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:10.965374 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.965827 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.966830 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:10.967456 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:10.967530 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:10.969400 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:10.969405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:10.969411 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:10.969438 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:10.970710 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:10.971130 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:10.971137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:10.971138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:10.971203 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:10.971473 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:10.971476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:10.971478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:10.971631 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:10.971726 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:10.971990 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 02:57:10.972088 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:57:10.973079 3272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:57:10.973765 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.973487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:42659 (local address 127.0.0.1:47664)
0925 02:57:10.973605 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:10.973652 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 02:57:10.974586 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.974369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:42659 (local address 127.0.0.1:47676)
0925 02:57:10.974495 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:10.974537 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260925 02:57:10.974709 3272 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.252:42659: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
I20260925 02:57:10.999609 3347 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } attempt: 1
I20260925 02:57:11.000121 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:10.999921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:42659 (local address 127.0.0.1:47680)
0925 02:57:11.000029 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:11.000074 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 02:57:11.000767 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.000569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:42659 (local address 127.0.0.1:47696)
0925 02:57:11.000663 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:11.000707 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 02:57:11.000929 3347 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.252:42659: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
I20260925 02:57:11.005916 3272 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } attempt: 1
I20260925 02:57:11.006451 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.006240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:42659 (local address 127.0.0.1:47704)
0925 02:57:11.006343 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:11.006390 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 02:57:11.007223 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.006953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:42659 (local address 127.0.0.1:47714)
0925 02:57:11.007100 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:11.007153 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 02:57:11.007375 3272 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.252:42659: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
W20260925 02:57:11.049423 3353 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:57:11.049696 3353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:11.049767 3353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:11.049873 3353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:11.054128 3353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:11.054175 3353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:11.054194 3353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:11.054247 3353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:11.054280 3353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:11.058976 3353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:38519,127.12.222.253:44223,127.12.222.252:42659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.252.keytab
--principal=kudu/127.12.222.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.252:42659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.222.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:57:11.059824 3353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:11.060668 3353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:11.068064 3360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:11.068058 3362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:11.068125 3353 server_base.cc:1034] running on GCE node
W20260925 02:57:11.068056 3359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:11.068781 3353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:11.069077 3347 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } attempt: 2
I20260925 02:57:11.069702 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.069445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:42659 (local address 127.0.0.1:47722)
0925 02:57:11.069574 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:11.069629 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 02:57:11.069921 3353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:11.070341 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.070146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:42659 (local address 127.0.0.1:47726)
0925 02:57:11.070248 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:11.070293 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 02:57:11.070456 3347 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.252:42659: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
I20260925 02:57:11.071089 3353 hybrid_clock.cc:648] HybridClock initialized: now 1790305031071062 us; error 35 us; skew 500 ppm
Sep 25 02:57:11 dist-test-slave-252r krb5kdc[3191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305031, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:11.073767 3353 init.cc:377] Logged in from keytab as kudu/127.12.222.252@KRBTEST.COM (short username kudu)
I20260925 02:57:11.074438 3353 webserver.cc:460] Webserver started at http://127.12.222.252:35373/ using document root <none> and password file <none>
I20260925 02:57:11.074900 3353 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:11.074952 3353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:11.075163 3353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:11.076977 3353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-2/data/instance:
uuid: "df1cfcb15157487a833c72d7140c33b4"
format_stamp: "Formatted at 2026-09-25 02:57:11 on dist-test-slave-252r"
server_key: "f3763a5a4ed96c7463ead6bc7adad104"
server_key_iv: "0fb84d8ce58410f5b46666d1ef8b36fd"
server_key_version: "encryptionkey@0"
I20260925 02:57:11.077523 3353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-2/wal/instance:
uuid: "df1cfcb15157487a833c72d7140c33b4"
format_stamp: "Formatted at 2026-09-25 02:57:11 on dist-test-slave-252r"
server_key: "f3763a5a4ed96c7463ead6bc7adad104"
server_key_iv: "0fb84d8ce58410f5b46666d1ef8b36fd"
server_key_version: "encryptionkey@0"
I20260925 02:57:11.081090 3353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:57:11.082522 3272 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } attempt: 2
I20260925 02:57:11.083180 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.082921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:42659 (local address 127.0.0.1:47728)
0925 02:57:11.083066 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:11.083121 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 02:57:11.083323 3369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:11.083793 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.083608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:42659 (local address 127.0.0.1:47734)
0925 02:57:11.083703 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:11.083743 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 02:57:11.083976 3272 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.252:42659: Network error: Client connection negotiation failed: client connection to 127.12.222.252:42659: connect: Connection refused (error 111)
I20260925 02:57:11.084583 3353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:57:11.084668 3353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-2/wal
uuid: "df1cfcb15157487a833c72d7140c33b4"
format_stamp: "Formatted at 2026-09-25 02:57:11 on dist-test-slave-252r"
server_key: "f3763a5a4ed96c7463ead6bc7adad104"
server_key_iv: "0fb84d8ce58410f5b46666d1ef8b36fd"
server_key_version: "encryptionkey@0"
I20260925 02:57:11.084736 3353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:11.099763 3353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:11.102155 3353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:11.102276 3353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:11.108314 3353 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.252:42659
I20260925 02:57:11.108572 3421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.252:42659 every 8 connection(s)
I20260925 02:57:11.109153 3353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-2/data/info.pb
I20260925 02:57:11.110527 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 3353
I20260925 02:57:11.110649 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304762911399-13179-0/minicluster-data/master-2/wal/instance
I20260925 02:57:11.110922 13179 external_mini_cluster.cc:1428] Setting key d95c107064f3465e49c0fc9650f0fb2e
I20260925 02:57:11.112046 3422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:11.116293 3422 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:57:11 dist-test-slave-252r krb5kdc[3191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.252@KRBTEST.COM
Sep 25 02:57:11 dist-test-slave-252r krb5kdc[3191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305031, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.252@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:57:11.125656 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.112083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47750 (local address 127.12.222.252:42659)
0925 02:57:11.112524 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.112546 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.112583 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:57:11.113344 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.113366 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.113742 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.114064 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.114675 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.116047 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.117006 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.117660 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.118346 (+ 686us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.120784 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:11.120789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:11.120797 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:11.120821 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:11.122578 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.123045 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.123071 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.123079 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.123225 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.123488 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.123492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.123494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.123916 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:11.124108 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:11.124424 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 02:57:11.124718 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":142,"threads_started":1}
I20260925 02:57:11.127790 3426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.118019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:38519 (local address 127.0.0.1:34754)
0925 02:57:11.118794 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:11.118844 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:11.118953 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:11.119228 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:11.119235 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:11.119684 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:11.119804 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:11.119810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.120682 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.120686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:11.121442 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:11.121452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.121903 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.121907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:11.122019 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.122475 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:11.122486 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:11.124024 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:11.125903 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:11.125910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:11.125915 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:11.126195 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:11.126587 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:11.126592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:11.126597 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:11.126701 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:11.127221 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:11.127226 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:11.127324 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 02:57:11.127453 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 02:57:11.127616 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":152,"threads_started":1}
I20260925 02:57:11.127846 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.118863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34754 (local address 127.12.222.254:38519)
0925 02:57:11.118978 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.118982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.119012 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:57:11.119047 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.119049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.119086 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.119143 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.119939 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.120559 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.121567 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.121790 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.121905 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.124163 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:11.124166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:11.124168 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:11.124191 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:11.125777 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.126331 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.126338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.126340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.126457 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.126821 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.126827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.126829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.127066 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:11.127148 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:11.127542 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 02:57:11.127662 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:57:11.128818 3422 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:57:11 dist-test-slave-252r krb5kdc[3191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
Sep 25 02:57:11 dist-test-slave-252r krb5kdc[3191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305031, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.252@KRBTEST.COM for kudu/127.12.222.253@KRBTEST.COM
I20260925 02:57:11.136077 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.128114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34756 (local address 127.12.222.254:38519)
0925 02:57:11.128397 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.128402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.128415 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:57:11.128446 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.128449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.128486 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.128562 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.129143 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.129731 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.130437 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.130642 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.130736 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.133116 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:11.133119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:11.133122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:11.133145 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:11.134557 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.134972 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.134975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.134976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.135053 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.135316 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.135319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.135321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.135503 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:11.135724 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:11.135839 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 02:57:11.135938 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 02:57:11.137758 3426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.129237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:44223 (local address 127.0.0.1:51588)
0925 02:57:11.129866 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:11.129876 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:11.129970 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:11.130191 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:11.130193 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:11.130353 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:11.130428 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:11.130433 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.131139 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.131159 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:11.132032 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:11.132042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.132549 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.132553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:11.132652 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.133081 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:11.133094 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:11.134397 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:11.136174 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:11.136178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:11.136182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:11.136418 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:11.136733 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:11.136737 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:11.136739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:11.136821 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:11.137278 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:11.137283 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:11.137396 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 02:57:11.137503 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:57:11.137618 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574}
I20260925 02:57:11.137796 3352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.129360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.12.222.253:44223)
0925 02:57:11.129488 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.129492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.129940 (+ 448us) server_negotiation.cc:407] Connection header received
0925 02:57:11.130045 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.130047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.130091 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.130180 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.130531 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.131022 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.132157 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.132436 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.132548 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.134523 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:11.134526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:11.134529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:11.134570 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:11.135949 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.136521 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.136524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.136526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.136603 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.136925 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.136928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.136930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.137130 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:11.137228 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:11.137568 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 02:57:11.137668 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:57:11.138592 3422 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:57:11 dist-test-slave-252r krb5kdc[3191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.253@KRBTEST.COM
Sep 25 02:57:11 dist-test-slave-252r krb5kdc[3191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305031, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.252@KRBTEST.COM for kudu/127.12.222.252@KRBTEST.COM
I20260925 02:57:11.146296 3427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.137189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.12.222.253:44223)
0925 02:57:11.137538 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.137545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.137555 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:57:11.137582 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.137584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.137613 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.137707 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.138314 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.139036 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.139989 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.140339 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.140457 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.142898 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:11.142909 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:11.142911 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:11.142931 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:11.144645 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.145085 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.145092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.145094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.145179 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.145428 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.145434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.145438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.145640 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:11.145822 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:11.145979 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 02:57:11.146121 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":145,"threads_started":1}
I20260925 02:57:11.146687 3426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.138997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:42659 (local address 127.0.0.1:47752)
0925 02:57:11.139126 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:11.139138 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:11.139204 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:11.139393 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:11.139402 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:11.139550 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:11.139624 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:11.139630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.140464 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.140467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:11.141251 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:11.141265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.141691 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.141694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:11.141794 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.142229 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:11.142239 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:11.143360 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:11.145129 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:11.145134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:11.145139 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:11.145359 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:11.145628 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:11.145631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:11.145633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:11.145722 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:11.146278 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:11.146280 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:11.146374 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 02:57:11.146481 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:57:11.146594 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 02:57:11.147163 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.139116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47752 (local address 127.12.222.252:42659)
0925 02:57:11.139204 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.139207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.139221 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:57:11.139273 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.139275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.139304 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.139359 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.139724 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.140366 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.141390 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.141590 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.141659 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.143502 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:11.143508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:11.143510 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:11.143534 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:11.145026 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.145469 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.145472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.145473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.145538 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.145812 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.145816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.145817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.146053 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:11.146878 (+ 825us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:11.146955 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 02:57:11.147055 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 02:57:11.149642 3422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4: Bootstrap starting.
I20260925 02:57:11.149653 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:57:11.151682 3422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:11.150874 3246 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9ec9563c17704f6987a2a1e09a12368d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34756
I20260925 02:57:11.152451 3422 log.cc:824] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:11.154101 3422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4: No bootstrap required, opened a new log
I20260925 02:57:11.156801 3422 raft_consensus.cc:348] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } } peers { permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } } peers { permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } }
I20260925 02:57:11.157003 3422 raft_consensus.cc:729] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1cfcb15157487a833c72d7140c33b4, State: Initialized, Role: FOLLOWER
I20260925 02:57:11.157295 3422 consensus_queue.cc:260] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } } peers { permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } } peers { permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } }
I20260925 02:57:11.157725 3429 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } } peers { permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } } peers { permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } } }
I20260925 02:57:11.157846 3429 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:11.158289 3422 sys_catalog.cc:564] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:11.161595 3347 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } attempt: 3
W20260925 02:57:11.162676 3440 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:57:11.162827 3440 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:57:11.161485 3353 master_runner.cc:186] Error getting master registration for 127.12.222.254:38519: OK, instance_id { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" instance_seqno: 1790305030725500 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:57:11.164621 3353 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 25 02:57:11 dist-test-slave-252r krb5kdc[3191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305030, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.253@KRBTEST.COM for kudu/127.12.222.252@KRBTEST.COM
I20260925 02:57:11.169602 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.162329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47754 (local address 127.12.222.252:42659)
0925 02:57:11.162422 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.162425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.162436 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:57:11.162463 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.162465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.162497 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.162944 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.163017 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.163658 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.164669 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.164898 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.165000 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.166759 (+ 1759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:11.166762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:11.166765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:11.166787 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:11.168173 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.168594 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.168599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.168600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.168669 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.168926 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.168928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.168930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.169068 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:11.169163 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:11.169390 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 02:57:11.169477 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 02:57:11.169602 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.161923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:42659 (local address 127.0.0.1:47754)
0925 02:57:11.162140 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:11.162152 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:11.162216 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:11.162605 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:11.162608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:11.162766 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:11.162846 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:11.162852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.163797 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.163800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:11.164533 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:11.164546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.165004 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.165007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:11.165078 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.165504 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:11.165517 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:11.166631 (+ 1114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:11.168308 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:11.168311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:11.168313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:11.168494 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:11.168753 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:11.168757 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:11.168758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:11.168838 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:11.169158 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:11.169161 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:11.169247 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 02:57:11.169343 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 02:57:11.169479 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260925 02:57:11.171816 3347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77: Bootstrap starting.
I20260925 02:57:11.173714 3347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:11.174321 3347 log.cc:824] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:11.175817 3347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77: No bootstrap required, opened a new log
I20260925 02:57:11.178527 3347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } } peers { permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } } peers { permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } }
I20260925 02:57:11.178704 3347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1f9b4fd1ed4fbf9eb12fa384a68e77, State: Initialized, Role: FOLLOWER
I20260925 02:57:11.178970 3347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } } peers { permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } } peers { permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } }
I20260925 02:57:11.179358 3443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } } peers { permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } } peers { permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } } }
I20260925 02:57:11.179472 3443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:11.179989 3347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:11.183107 3272 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } attempt: 3
W20260925 02:57:11.184558 3454 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:57:11.184649 3454 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:57:11.183243 3279 master_runner.cc:186] Error getting master registration for 127.12.222.254:38519: OK, instance_id { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" instance_seqno: 1790305030725500 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:57:11.186430 3279 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 25 02:57:11 dist-test-slave-252r krb5kdc[3191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305030, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for kudu/127.12.222.252@KRBTEST.COM
I20260925 02:57:11.191799 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.183861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47768 (local address 127.12.222.252:42659)
0925 02:57:11.183985 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.183988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.183999 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:57:11.184030 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.184032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.184065 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.184125 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.184640 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.185286 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.186304 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.186641 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.186741 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.188777 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:11.188780 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:11.188782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:11.188802 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:11.190283 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.190745 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.190748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.190750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.190839 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.191100 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.191103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.191105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.191266 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:11.191364 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:11.191588 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 02:57:11.191676 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:57:11.191799 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.183455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:42659 (local address 127.0.0.1:47768)
0925 02:57:11.183667 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:11.183680 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:11.183753 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:11.184220 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:11.184223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:11.184413 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:11.184515 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:11.184521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.185419 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.185422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:11.186162 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:11.186173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.186752 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.186756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:11.186841 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.187294 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:11.187326 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:11.188669 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:11.190419 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:11.190422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:11.190424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:11.190653 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:11.190933 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:11.190937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:11.190938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:11.191025 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:11.191360 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:11.191363 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:11.191454 (+ 91us) client_negotiation.cc:769] Sending connection context
0925 02:57:11.191543 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 02:57:11.191662 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260925 02:57:11.194689 3272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d: Bootstrap starting.
I20260925 02:57:11.197114 3272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:11.197822 3272 log.cc:824] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:11.199684 3272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d: No bootstrap required, opened a new log
I20260925 02:57:11.202464 3272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } } peers { permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } } peers { permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } }
I20260925 02:57:11.202651 3272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec9563c17704f6987a2a1e09a12368d, State: Initialized, Role: FOLLOWER
I20260925 02:57:11.202961 3272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } } peers { permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } } peers { permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } }
I20260925 02:57:11.203352 3457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } } peers { permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } } peers { permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } } }
I20260925 02:57:11.203568 3457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:11.204057 3272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:11.204101 3246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 02:57:11.204177 3246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:11.205471 3246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } } peers { permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } } peers { permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } }
I20260925 02:57:11.206907 3321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ec9563c17704f6987a2a1e09a12368d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77"
I20260925 02:57:11.207124 3321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:11.207095 3396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ec9563c17704f6987a2a1e09a12368d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "df1cfcb15157487a833c72d7140c33b4"
I20260925 02:57:11.207309 3396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:11.208324 3321 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ec9563c17704f6987a2a1e09a12368d in term 1.
I20260925 02:57:11.209053 3223 leader_election.cc:302] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a1f9b4fd1ed4fbf9eb12fa384a68e77, 9ec9563c17704f6987a2a1e09a12368d; no voters:
I20260925 02:57:11.209125 3396 raft_consensus.cc:2454] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ec9563c17704f6987a2a1e09a12368d in term 1.
W20260925 02:57:11.209681 3468 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:57:11.209764 3468 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:57:11.209936 3457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:11.210224 3457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [term 1 LEADER]: Becoming Leader. State: Replica: 9ec9563c17704f6987a2a1e09a12368d, State: Running, Role: LEADER
I20260925 02:57:11.210549 3457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } } peers { permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } } peers { permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } }
I20260925 02:57:11.212248 3457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ec9563c17704f6987a2a1e09a12368d. Latest consensus state: current_term: 1 leader_uuid: "9ec9563c17704f6987a2a1e09a12368d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } } peers { permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } } peers { permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } } }
I20260925 02:57:11.212411 3457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:11.212790 3472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:11.213649 3472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:11.213984 3204 master_runner.cc:263] No leader master found from master 9ec9563c17704f6987a2a1e09a12368d
I20260925 02:57:11.214046 3204 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:57:11.206406 3246 leader_election.cc:288] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [CANDIDATE]: Term 1 election: Requested vote from peers 2a1f9b4fd1ed4fbf9eb12fa384a68e77 (127.12.222.253:44223), df1cfcb15157487a833c72d7140c33b4 (127.12.222.252:42659)
W20260925 02:57:11.222543 3225 tablet.cc:2369] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:57:11.224043 3321 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77 [term 1 FOLLOWER]: Refusing update from remote peer 9ec9563c17704f6987a2a1e09a12368d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:11.224043 3396 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4 [term 1 FOLLOWER]: Refusing update from remote peer 9ec9563c17704f6987a2a1e09a12368d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:11.224704 3457 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [LEADER]: Connected to new peer: Peer: permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:11.224967 3481 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:11.228410 3429 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ec9563c17704f6987a2a1e09a12368d. Latest consensus state: current_term: 1 leader_uuid: "9ec9563c17704f6987a2a1e09a12368d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } } peers { permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } } peers { permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } } }
I20260925 02:57:11.228513 3429 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:11.228622 3443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ec9563c17704f6987a2a1e09a12368d. Latest consensus state: current_term: 1 leader_uuid: "9ec9563c17704f6987a2a1e09a12368d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } } peers { permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } } peers { permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } } }
I20260925 02:57:11.228754 3443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:11.248809 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.222577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34760 (local address 127.12.222.254:38519)
0925 02:57:11.222676 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.222680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.240508 (+ 17828us) server_negotiation.cc:407] Connection header received
0925 02:57:11.240619 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.240622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.240665 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.240726 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.241455 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.242203 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.243273 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.243585 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.243664 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.245136 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:11.245142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:11.245147 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:11.245172 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:11.246863 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.247316 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.247319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.247321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.247419 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.247691 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.247694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.247695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.247870 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:11.247976 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:11.248564 (+ 588us) server_negotiation.cc:299] Negotiation successful
0925 02:57:11.248666 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:57:11.259356 3477 mvcc.cc:205] Tried to move back new op lower bound from 7333089407882514432 to 7333089407841898496. Current Snapshot: MvccSnapshot[applied={T|T < 7333089407882514432}]
I20260925 02:57:11.259760 3470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ec9563c17704f6987a2a1e09a12368d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } } peers { permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } } peers { permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } } }
I20260925 02:57:11.259900 3470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:11.263726 3427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.218923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51614 (local address 127.12.222.253:44223)
0925 02:57:11.219036 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.219041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.228831 (+ 9790us) server_negotiation.cc:407] Connection header received
0925 02:57:11.228871 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.228874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.228925 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.229005 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.231324 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.232069 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.234658 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.235098 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.235192 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.252549 (+ 17357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:11.252553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:11.252555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:11.252580 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:11.254280 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.261195 (+ 6915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.261198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.261200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.261278 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.262070 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.262074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.262075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.262261 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:11.262366 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:11.263465 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0925 02:57:11.263576 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:57:11.263695 3457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ec9563c17704f6987a2a1e09a12368d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } } peers { permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } } peers { permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } } }
I20260925 02:57:11.263784 3457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:11.264819 3429 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ec9563c17704f6987a2a1e09a12368d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } } peers { permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } } peers { permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } } }
I20260925 02:57:11.264930 3429 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:11.265094 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:11.219686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47770 (local address 127.12.222.252:42659)
0925 02:57:11.219789 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.219793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.230215 (+ 10422us) server_negotiation.cc:407] Connection header received
0925 02:57:11.230356 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.230359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.230410 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.230453 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.233196 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.233837 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.236213 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:11.236588 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:11.236657 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:11.254769 (+ 18112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:11.254773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:11.254775 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:11.254800 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:11.256409 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.261710 (+ 5301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.261713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.261715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.261792 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:11.262642 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:11.262646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:11.262647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:11.262827 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:11.262920 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:11.264863 (+ 1943us) server_negotiation.cc:299] Negotiation successful
0925 02:57:11.264961 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:57:11.266330 3472 catalog_manager.cc:1347] Generated new cluster ID: 2ecc2e6957ef40b3b4dd037ff06f677e
I20260925 02:57:11.266396 3472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:11.276103 3482 mvcc.cc:205] Tried to move back new op lower bound from 7333089407882514432 to 7333089407841898496. Current Snapshot: MvccSnapshot[applied={T|T < 7333089407882514432 or (T in {7333089407882514432})}]
I20260925 02:57:11.277468 3472 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:11.278478 3472 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:11.278992 3443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ec9563c17704f6987a2a1e09a12368d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ec9563c17704f6987a2a1e09a12368d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 38519 } } peers { permanent_uuid: "2a1f9b4fd1ed4fbf9eb12fa384a68e77" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 44223 } } peers { permanent_uuid: "df1cfcb15157487a833c72d7140c33b4" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 42659 } } }
I20260925 02:57:11.279075 3443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:11.279826 3483 mvcc.cc:205] Tried to move back new op lower bound from 7333089407882514432 to 7333089407841898496. Current Snapshot: MvccSnapshot[applied={T|T < 7333089407882514432}]
I20260925 02:57:11.299573 3472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ec9563c17704f6987a2a1e09a12368d: Generated new TSK 0
I20260925 02:57:11.300261 3472 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:57:11.325289 3427 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:11.322842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51620 (local address 127.12.222.253:44223)
0925 02:57:11.322962 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.322966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.324228 (+ 1262us) server_negotiation.cc:407] Connection header received
0925 02:57:11.324469 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.324473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.324522 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.324574 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.325174 (+ 600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51620: BlockingRecv error: recv got EOF from 127.0.0.1:51620 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 02:57:11.326602 3425 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:11.323653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47786 (local address 127.12.222.252:42659)
0925 02:57:11.323770 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.323774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.325725 (+ 1951us) server_negotiation.cc:407] Connection header received
0925 02:57:11.325868 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.325870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.325916 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.325962 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.326491 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47786: BlockingRecv error: recv got EOF from 127.0.0.1:47786 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 02:57:11.328478 3275 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:11.327205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.12.222.254:38519)
0925 02:57:11.327326 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.327330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.327627 (+ 297us) server_negotiation.cc:407] Connection header received
0925 02:57:11.327794 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.327797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.327844 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.327894 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.328366 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34772: BlockingRecv error: recv got EOF from 127.0.0.1:34772 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 02:57:11.331972 3427 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:11.330755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51624 (local address 127.12.222.253:44223)
0925 02:57:11.330836 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.330839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.331241 (+ 402us) server_negotiation.cc:407] Connection header received
0925 02:57:11.331411 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.331413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.331469 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.331510 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.331893 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51624: BlockingRecv error: recv got EOF from 127.0.0.1:51624 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260925 02:57:11.336272 3427 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:11.334885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51632 (local address 127.12.222.253:44223)
0925 02:57:11.334971 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.334975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.335457 (+ 482us) server_negotiation.cc:407] Connection header received
0925 02:57:11.335636 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.335639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.335679 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.335723 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.336167 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51632: BlockingRecv error: recv got EOF from 127.0.0.1:51632 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 02:57:11.341296 3427 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:11.339909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51634 (local address 127.12.222.253:44223)
0925 02:57:11.340003 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.340008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.340517 (+ 509us) server_negotiation.cc:407] Connection header received
0925 02:57:11.340674 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.340677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.340714 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.340755 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.341222 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51634: BlockingRecv error: recv got EOF from 127.0.0.1:51634 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260925 02:57:11.348057 3427 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:11.347017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51636 (local address 127.12.222.253:44223)
0925 02:57:11.347140 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.347144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.347379 (+ 235us) server_negotiation.cc:407] Connection header received
0925 02:57:11.347497 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.347500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.347539 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.347639 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.347986 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51636: BlockingRecv error: recv got EOF from 127.0.0.1:51636 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 02:57:11.358407 3427 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:11.357470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51648 (local address 127.12.222.253:44223)
0925 02:57:11.357565 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.357568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.357780 (+ 212us) server_negotiation.cc:407] Connection header received
0925 02:57:11.357881 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.357884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.357925 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.357997 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.358330 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51648: BlockingRecv error: recv got EOF from 127.0.0.1:51648 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 02:57:11.376929 3427 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:11.375837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51660 (local address 127.12.222.253:44223)
0925 02:57:11.375938 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.375941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.376166 (+ 225us) server_negotiation.cc:407] Connection header received
0925 02:57:11.376342 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.376345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.376390 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.376472 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.376832 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51660: BlockingRecv error: recv got EOF from 127.0.0.1:51660 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 02:57:11.411623 3427 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:11.410476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.12.222.253:44223)
0925 02:57:11.410609 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.410616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.410803 (+ 187us) server_negotiation.cc:407] Connection header received
0925 02:57:11.410965 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.410968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.411027 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.411104 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.411528 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51666: BlockingRecv error: recv got EOF from 127.0.0.1:51666 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 02:57:11.478721 3427 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:11.477550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51678 (local address 127.12.222.253:44223)
0925 02:57:11.477721 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.477725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.477877 (+ 152us) server_negotiation.cc:407] Connection header received
0925 02:57:11.478040 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.478044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.478096 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.478185 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.478627 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51678: BlockingRecv error: recv got EOF from 127.0.0.1:51678 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 02:57:11.610013 3427 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:11.608839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.12.222.253:44223)
0925 02:57:11.609009 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.609015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.609111 (+ 96us) server_negotiation.cc:407] Connection header received
0925 02:57:11.609235 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.609237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.609285 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.609356 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.609863 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51680: BlockingRecv error: recv got EOF from 127.0.0.1:51680 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260925 02:57:11.869155 3427 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:11.868007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.12.222.253:44223)
0925 02:57:11.868144 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:11.868149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:11.868247 (+ 98us) server_negotiation.cc:407] Connection header received
0925 02:57:11.868440 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:11.868443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:11.868486 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:11.868570 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:11.869022 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51696: BlockingRecv error: recv got EOF from 127.0.0.1:51696 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:57:12.164036 3440 catalog_manager.cc:1259] Loaded cluster ID: 2ecc2e6957ef40b3b4dd037ff06f677e
I20260925 02:57:12.164106 3440 catalog_manager.cc:1552] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4: loading cluster ID for follower catalog manager: success
I20260925 02:57:12.165663 3440 catalog_manager.cc:1574] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4: acquiring CA information for follower catalog manager: success
I20260925 02:57:12.166738 3440 catalog_manager.cc:1602] T 00000000000000000000000000000000 P df1cfcb15157487a833c72d7140c33b4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:57:12.185652 3454 catalog_manager.cc:1259] Loaded cluster ID: 2ecc2e6957ef40b3b4dd037ff06f677e
I20260925 02:57:12.185701 3454 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77: loading cluster ID for follower catalog manager: success
I20260925 02:57:12.187136 3454 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77: acquiring CA information for follower catalog manager: success
I20260925 02:57:12.188222 3454 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2a1f9b4fd1ed4fbf9eb12fa384a68e77: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:57:12.386322 3553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:12.383034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.12.222.253:44223)
0925 02:57:12.383241 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:12.383245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:12.383315 (+ 70us) server_negotiation.cc:407] Connection header received
0925 02:57:12.383430 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:12.383433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:12.383564 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:12.383660 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:12.383877 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.384363 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.385079 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.385264 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.385326 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:12.385564 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:12.385952 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:12.386033 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:12.386073 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 02:57:12.386113 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
I20260925 02:57:12.392174 3559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:12.389242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.12.222.252:42659)
0925 02:57:12.389448 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:12.389451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:12.389463 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:57:12.389533 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:12.389535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:12.389629 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:12.389714 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:12.389883 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.390323 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.390992 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.391184 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.391249 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:12.391466 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:12.391795 (+ 329us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:12.391885 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:12.391929 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 02:57:12.391970 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":61,"threads_started":1}
I20260925 02:57:12.398991 3559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:12.394949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.12.222.252:42659)
0925 02:57:12.395077 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:12.395080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:12.395396 (+ 316us) server_negotiation.cc:407] Connection header received
0925 02:57:12.395559 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:12.395562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:12.395611 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:12.395653 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:12.396140 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.396722 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.397952 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.398259 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.398350 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:12.398567 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:12.398633 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:12.398708 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:12.398835 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 02:57:12.398880 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:57:12.399225 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:12.394950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34788 (local address 127.12.222.254:38519)
0925 02:57:12.395538 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:12.395543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:12.395558 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:57:12.395666 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:12.395669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:12.395766 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:12.395926 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:12.396219 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.397030 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.397832 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.398174 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.398273 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:12.398571 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:12.398841 (+ 270us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:12.398921 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:12.399012 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 02:57:12.399063 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":97,"threads_started":1}
I20260925 02:57:12.399509 3553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:12.394825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51702 (local address 127.12.222.253:44223)
0925 02:57:12.394940 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:12.394943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:12.395323 (+ 380us) server_negotiation.cc:407] Connection header received
0925 02:57:12.395521 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:12.395527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:12.395578 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:12.395622 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:12.396084 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.396893 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.398416 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:12.398704 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:12.398805 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:12.399068 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:12.399145 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:12.399192 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:12.399353 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 02:57:12.399400 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 02:57:12.405441 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 3204
I20260925 02:57:12.411545 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 3279
I20260925 02:57:12.417253 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 3353
2026-09-25T02:57:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2929 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:57:12 dist-test-slave-252r krb5kdc[3570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:57:12 dist-test-slave-252r krb5kdc[3570](info): set up 2 sockets
Sep 25 02:57:12 dist-test-slave-252r krb5kdc[3570](info): commencing operation
krb5kdc: starting...
W20260925 02:57:13.384545 13179 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/test-user.keytab.
Sep 25 02:57:13 dist-test-slave-252r krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:57:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:57:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.222.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
Entry for principal kudu/127.12.222.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab.
I20260925 02:57:13.490617 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:33469
--webserver_interface=127.12.222.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:33469,127.12.222.253:46739,127.12.222.252:44459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:57:13.593539 3583 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:57:13.593760 3583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:13.593781 3583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:13.593863 3583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:13.597625 3583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:13.597671 3583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:13.597709 3583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:13.597741 3583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:13.597766 3583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:13.602510 3583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:33469,127.12.222.253:46739,127.12.222.252:44459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.254.keytab
--principal=kudu/127.12.222.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.254:33469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.222.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:57:13.603368 3583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:13.604324 3583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:57:13.612089 3591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:13.612106 3588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:13.612049 3589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:13.612195 3583 server_base.cc:1034] running on GCE node
I20260925 02:57:13.612757 3583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:13.613642 3583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:13.614820 3583 hybrid_clock.cc:648] HybridClock initialized: now 1790305033614803 us; error 32 us; skew 500 ppm
Sep 25 02:57:13 dist-test-slave-252r krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305033, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:13.617437 3583 init.cc:377] Logged in from keytab as kudu/127.12.222.254@KRBTEST.COM (short username kudu)
I20260925 02:57:13.618224 3583 webserver.cc:460] Webserver started at http://127.12.222.254:33897/ using document root <none> and password file <none>
I20260925 02:57:13.618691 3583 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:13.618773 3583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:13.618985 3583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:13.620873 3583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-0/data/instance:
uuid: "59596ca58a994f98a4ca4a1cd2a9a71d"
format_stamp: "Formatted at 2026-09-25 02:57:13 on dist-test-slave-252r"
server_key: "2677c7e676d0516a21a2ee4ac145ebe9"
server_key_iv: "e592fe46a99eed4015f72251b5ebe012"
server_key_version: "encryptionkey@0"
I20260925 02:57:13.621414 3583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-0/wal/instance:
uuid: "59596ca58a994f98a4ca4a1cd2a9a71d"
format_stamp: "Formatted at 2026-09-25 02:57:13 on dist-test-slave-252r"
server_key: "2677c7e676d0516a21a2ee4ac145ebe9"
server_key_iv: "e592fe46a99eed4015f72251b5ebe012"
server_key_version: "encryptionkey@0"
I20260925 02:57:13.624604 3583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:57:13.626662 3598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:13.627749 3583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:57:13.627880 3583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-0/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-0/wal
uuid: "59596ca58a994f98a4ca4a1cd2a9a71d"
format_stamp: "Formatted at 2026-09-25 02:57:13 on dist-test-slave-252r"
server_key: "2677c7e676d0516a21a2ee4ac145ebe9"
server_key_iv: "e592fe46a99eed4015f72251b5ebe012"
server_key_version: "encryptionkey@0"
I20260925 02:57:13.627987 3583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:13.639101 3583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:13.641952 3583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:13.642143 3583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:13.648356 3583 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.254:33469
I20260925 02:57:13.648617 3650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.254:33469 every 8 connection(s)
I20260925 02:57:13.649153 3583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-0/data/info.pb
I20260925 02:57:13.651554 3651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:13.654656 3651 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:57:13.655392 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 3583
I20260925 02:57:13.655516 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-0/wal/instance
I20260925 02:57:13.655792 13179 external_mini_cluster.cc:1428] Setting key 0c5dedcc5cfa7b400b88c460eb6fc1c3
WARNING: no policy specified for kudu/127.12.222.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.253@KRBTEST.COM" created.
Sep 25 02:57:13 dist-test-slave-252r krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305033, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:57:13.672940 3653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.656855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.12.222.254:33469)
0925 02:57:13.657309 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:13.657329 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:13.657585 (+ 256us) server_negotiation.cc:407] Connection header received
0925 02:57:13.658468 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:13.658476 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:13.658684 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:13.658856 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:13.659514 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:13.660983 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:13.661913 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:13.662494 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:13.662672 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:13.667407 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:13.667417 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:13.667431 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:13.667460 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:13.669427 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:13.669854 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:13.669861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:13.669865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:13.669994 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:13.670381 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:13.670384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:13.670386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:13.670878 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:13.671015 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:13.672313 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0925 02:57:13.672460 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":294,"thread_start_us":132,"threads_started":1}
I20260925 02:57:13.673352 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.656855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:33469 (local address 127.0.0.1:40108)
0925 02:57:13.657416 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:13.657459 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:13.658193 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:13.658868 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:13.658872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:13.659132 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:13.659358 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:13.659369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:13.661111 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:13.661114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:13.661795 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:13.661803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:13.662635 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:13.662638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:13.662709 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:13.663231 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:13.663246 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:13.667213 (+ 3967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:13.669538 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:13.669545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:13.669553 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:13.669751 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:13.670115 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:13.670118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:13.670121 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:13.670270 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:13.670992 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:13.670997 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:13.671131 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 02:57:13.671293 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 02:57:13.671577 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":3,"spinlock_wait_cycles":47488,"thread_start_us":148,"threads_started":1}
I20260925 02:57:13.675392 3651 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:57:13.676003 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.675783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:46739 (local address 127.0.0.1:52184)
0925 02:57:13.675881 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:13.675939 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.253:46739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 02:57:13.676190 3602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.222.253:46739: connect: Connection refused (error 111)
I20260925 02:57:13.677019 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.676792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:46739 (local address 127.0.0.1:52192)
0925 02:57:13.676930 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:13.676971 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.253:46739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 02:57:13.677147 3651 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.253:46739: Network error: Client connection negotiation failed: client connection to 127.12.222.253:46739: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.253.keytab.
Entry for principal kudu/127.12.222.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.253.keytab.
I20260925 02:57:13.681053 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.253:46739
--webserver_interface=127.12.222.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.253.keytab
--principal=kudu/127.12.222.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:33469,127.12.222.253:46739,127.12.222.252:44459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:57:13.726423 3651 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } attempt: 1
I20260925 02:57:13.727216 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.726922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:46739 (local address 127.0.0.1:52204)
0925 02:57:13.727075 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:13.727137 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.253:46739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 02:57:13.727921 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.727715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:46739 (local address 127.0.0.1:52220)
0925 02:57:13.727811 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:13.727857 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.253:46739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 02:57:13.728058 3651 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.253:46739: Network error: Client connection negotiation failed: client connection to 127.12.222.253:46739: connect: Connection refused (error 111)
W20260925 02:57:13.783998 3658 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:57:13.784215 3658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:13.784289 3658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:13.784334 3658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:13.788024 3658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:13.788064 3658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:13.788081 3658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:13.788096 3658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:13.788110 3658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:13.792728 3658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:33469,127.12.222.253:46739,127.12.222.252:44459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.253.keytab
--principal=kudu/127.12.222.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.253:46739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.222.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:57:13.793537 3658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:13.794389 3658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:57:13.796223 3651 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } attempt: 2
I20260925 02:57:13.797035 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.796713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:46739 (local address 127.0.0.1:52222)
0925 02:57:13.796883 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:13.796951 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.253:46739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 02:57:13.797729 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.797518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:46739 (local address 127.0.0.1:52230)
0925 02:57:13.797623 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:13.797672 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.253:46739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 02:57:13.797859 3651 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.253:46739: Network error: Client connection negotiation failed: client connection to 127.12.222.253:46739: connect: Connection refused (error 111)
W20260925 02:57:13.802529 3664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:13.802523 3666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:13.802567 3658 server_base.cc:1034] running on GCE node
W20260925 02:57:13.802523 3663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:13.803233 3658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:13.804128 3658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:13.805287 3658 hybrid_clock.cc:648] HybridClock initialized: now 1790305033805265 us; error 29 us; skew 500 ppm
Sep 25 02:57:13 dist-test-slave-252r krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305033, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:13.807924 3658 init.cc:377] Logged in from keytab as kudu/127.12.222.253@KRBTEST.COM (short username kudu)
I20260925 02:57:13.808894 3658 webserver.cc:460] Webserver started at http://127.12.222.253:37979/ using document root <none> and password file <none>
I20260925 02:57:13.809410 3658 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:13.809496 3658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:13.809703 3658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:13.811553 3658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-1/data/instance:
uuid: "a91f82d25dd64807a9bf9f8ca27a7f00"
format_stamp: "Formatted at 2026-09-25 02:57:13 on dist-test-slave-252r"
server_key: "adc2bd8b903b157415f9ae4b79ba8444"
server_key_iv: "7949aaf87dee60bcdaa8bff35210985c"
server_key_version: "encryptionkey@0"
I20260925 02:57:13.812093 3658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-1/wal/instance:
uuid: "a91f82d25dd64807a9bf9f8ca27a7f00"
format_stamp: "Formatted at 2026-09-25 02:57:13 on dist-test-slave-252r"
server_key: "adc2bd8b903b157415f9ae4b79ba8444"
server_key_iv: "7949aaf87dee60bcdaa8bff35210985c"
server_key_version: "encryptionkey@0"
I20260925 02:57:13.815395 3658 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:57:13.817561 3673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:13.818629 3658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:57:13.818748 3658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-1/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-1/wal
uuid: "a91f82d25dd64807a9bf9f8ca27a7f00"
format_stamp: "Formatted at 2026-09-25 02:57:13 on dist-test-slave-252r"
server_key: "adc2bd8b903b157415f9ae4b79ba8444"
server_key_iv: "7949aaf87dee60bcdaa8bff35210985c"
server_key_version: "encryptionkey@0"
I20260925 02:57:13.818840 3658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:13.848359 3658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:13.851037 3658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:13.851241 3658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:13.857402 3658 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.253:46739
I20260925 02:57:13.857693 3725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.253:46739 every 8 connection(s)
I20260925 02:57:13.858225 3658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-1/data/info.pb
I20260925 02:57:13.860699 3726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:13.863988 3726 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:57:13.866111 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 3658
I20260925 02:57:13.866285 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-1/wal/instance
I20260925 02:57:13.866649 13179 external_mini_cluster.cc:1428] Setting key 87e897a1ba113f5e3fd384615390ae6e
Sep 25 02:57:13 dist-test-slave-252r krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305033, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.253@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.222.252@KRBTEST.COM; defaulting to no policy
I20260925 02:57:13.876462 3653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.865585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.12.222.254:33469)
0925 02:57:13.865720 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:13.865725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:13.866518 (+ 793us) server_negotiation.cc:407] Connection header received
0925 02:57:13.867430 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:13.867436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:13.867511 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:13.867641 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:13.868703 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:13.869220 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:13.870269 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:13.870497 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:13.870603 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:13.873085 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:13.873088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:13.873090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:13.873113 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:13.874532 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:13.875046 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:13.875049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:13.875050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:13.875123 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:13.875409 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:13.875411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:13.875412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:13.875597 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:13.875665 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:13.876124 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 02:57:13.876213 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.222.252@KRBTEST.COM" created.
I20260925 02:57:13.877122 3728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.865919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:33469 (local address 127.0.0.1:40124)
0925 02:57:13.866327 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:13.866386 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:13.867207 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:13.867774 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:13.867780 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:13.868080 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:13.868554 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:13.868566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:13.869342 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:13.869346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:13.870125 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:13.870134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:13.870636 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:13.870639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:13.870787 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:13.871278 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:13.871293 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:13.872922 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:13.874633 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:13.874637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:13.874646 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:13.874947 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:13.875202 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:13.875204 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:13.875218 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:13.875335 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:13.875676 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:13.875680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:13.875921 (+ 241us) client_negotiation.cc:769] Sending connection context
0925 02:57:13.876059 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 02:57:13.876317 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":114,"threads_started":1}
I20260925 02:57:13.878381 3726 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:57:13 dist-test-slave-252r krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305033, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.253@KRBTEST.COM for kudu/127.12.222.253@KRBTEST.COM
I20260925 02:57:13.887086 3728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.878825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:46739 (local address 127.0.0.1:52234)
0925 02:57:13.878999 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:13.879013 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:13.879082 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:13.879472 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:13.879477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:13.879660 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:13.879790 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:13.879801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:13.880750 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:13.880756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:13.881708 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:13.881721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:13.882244 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:13.882249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:13.882360 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:13.882720 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:13.882734 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:13.883850 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:13.885746 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:13.885749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:13.885751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:13.885929 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:13.886232 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:13.886234 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:13.886235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:13.886315 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:13.886747 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:13.886749 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:13.886814 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 02:57:13.886910 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 02:57:13.886995 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 02:57:13.887318 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.878899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52234 (local address 127.12.222.253:46739)
0925 02:57:13.879139 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:13.879147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:13.879163 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:57:13.879195 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:13.879198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:13.879344 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:13.879438 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:13.879916 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:13.880632 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:13.881830 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:13.882138 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:13.882208 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:13.883971 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:13.883979 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:13.883986 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:13.884003 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:13.885633 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:13.886029 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:13.886034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:13.886036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:13.886135 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:13.886442 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:13.886445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:13.886446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:13.886660 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:13.886785 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:13.886969 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 02:57:13.887067 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
I20260925 02:57:13.888805 3726 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:57:13.889019 3651 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } attempt: 3
I20260925 02:57:13.889698 3728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.889188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44459 (local address 127.0.0.1:42710)
0925 02:57:13.889561 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:13.889634 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314}
W20260925 02:57:13.890687 3676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.222.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.252.keytab.
Entry for principal kudu/127.12.222.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.252.keytab.
I20260925 02:57:13.891470 3728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.891266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44459 (local address 127.0.0.1:42716)
0925 02:57:13.891392 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:13.891428 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260925 02:57:13.891628 3726 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.252:44459: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
I20260925 02:57:13.891705 13179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskORQp8j/build/debug/bin/kudu
/tmp/dist-test-taskORQp8j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.252:44459
--webserver_interface=127.12.222.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.252.keytab
--principal=kudu/127.12.222.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.222.212:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.222.254:33469,127.12.222.253:46739,127.12.222.252:44459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:57:13 dist-test-slave-252r krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305033, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for kudu/127.12.222.253@KRBTEST.COM
I20260925 02:57:13.898288 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.889441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52238 (local address 127.12.222.253:46739)
0925 02:57:13.889532 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:13.889535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:13.889553 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:57:13.889655 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:13.889657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:13.889710 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:13.889790 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:13.890560 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:13.891221 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:13.892427 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:13.893331 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:13.893447 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:13.895332 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:13.895335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:13.895337 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:13.895355 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:13.896759 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:13.897249 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:13.897251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:13.897252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:13.897318 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:13.897624 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:13.897626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:13.897627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:13.897759 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:13.897826 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:13.898130 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 02:57:13.898205 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:57:13.898473 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.889410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:46739 (local address 127.0.0.1:52238)
0925 02:57:13.889496 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:13.889511 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:13.889578 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:13.889806 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:13.889808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:13.889953 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:13.890447 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:13.890454 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:13.891336 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:13.891339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:13.892304 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:13.892318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:13.893444 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:13.893446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:13.893517 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:13.893930 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:13.893941 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:13.895214 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:13.896871 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:13.896876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:13.896878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:13.897151 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:13.897411 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:13.897416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:13.897420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:13.897532 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:13.897852 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:13.897856 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:13.897967 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 02:57:13.898099 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 02:57:13.898244 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 02:57:13.899149 3651 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:57:13.899770 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.899572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44459 (local address 127.0.0.1:42730)
0925 02:57:13.899678 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:13.899720 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 02:57:13.900552 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.900283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44459 (local address 127.0.0.1:42732)
0925 02:57:13.900436 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:13.900480 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260925 02:57:13.900731 3651 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.252:44459: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
I20260925 02:57:13.931896 3651 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } attempt: 1
I20260925 02:57:13.932493 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.932220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44459 (local address 127.0.0.1:42742)
0925 02:57:13.932380 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:13.932430 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 02:57:13.933097 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.932908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44459 (local address 127.0.0.1:42750)
0925 02:57:13.933001 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:13.933047 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 02:57:13.933221 3651 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.252:44459: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
I20260925 02:57:13.940869 3726 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } attempt: 1
I20260925 02:57:13.941417 3728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.941182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44459 (local address 127.0.0.1:42766)
0925 02:57:13.941307 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:13.941357 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 02:57:13.942060 3728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:13.941828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44459 (local address 127.0.0.1:42774)
0925 02:57:13.941967 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:13.942012 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260925 02:57:13.942184 3726 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.252:44459: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
W20260925 02:57:13.992703 3733 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:57:13.992962 3733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:57:13.993021 3733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:57:13.993106 3733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:57:13.996532 3733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:57:13.996595 3733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:57:13.996636 3733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:57:13.996672 3733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:57:13.996703 3733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:57:14.001271 3733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.222.212:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.222.254:33469,127.12.222.253:46739,127.12.222.252:44459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/krb5kdc/kudu_127.12.222.252.keytab
--principal=kudu/127.12.222.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.222.252:44459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.222.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:57:14.002121 3733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:57:14.002995 3733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:57:14.008394 3651 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } attempt: 2
I20260925 02:57:14.009063 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.008795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44459 (local address 127.0.0.1:42786)
0925 02:57:14.008918 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:14.008974 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 02:57:14.009815 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.009543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44459 (local address 127.0.0.1:42792)
0925 02:57:14.009689 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:14.009746 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260925 02:57:14.010082 3651 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.252:44459: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
I20260925 02:57:14.010319 3726 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } attempt: 2
W20260925 02:57:14.010653 3741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:57:14.010707 3739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:14.010840 3733 server_base.cc:1034] running on GCE node
W20260925 02:57:14.011020 3738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:57:14.011098 3728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.010765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44459 (local address 127.0.0.1:42808)
0925 02:57:14.010949 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:14.011014 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260925 02:57:14.011497 3733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:57:14.011790 3728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.011573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44459 (local address 127.0.0.1:42824)
0925 02:57:14.011698 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:14.011739 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 02:57:14.011914 3726 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.222.252:44459: Network error: Client connection negotiation failed: client connection to 127.12.222.252:44459: connect: Connection refused (error 111)
I20260925 02:57:14.012616 3733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:57:14.013795 3733 hybrid_clock.cc:648] HybridClock initialized: now 1790305034013786 us; error 31 us; skew 500 ppm
Sep 25 02:57:14 dist-test-slave-252r krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305034, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:57:14.016170 3733 init.cc:377] Logged in from keytab as kudu/127.12.222.252@KRBTEST.COM (short username kudu)
I20260925 02:57:14.016945 3733 webserver.cc:460] Webserver started at http://127.12.222.252:45637/ using document root <none> and password file <none>
I20260925 02:57:14.017458 3733 fs_manager.cc:359] Metadata directory not provided
I20260925 02:57:14.017568 3733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:57:14.017777 3733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:57:14.019495 3733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-2/data/instance:
uuid: "4a433c0a8a804f2fb2386b2d75196ab3"
format_stamp: "Formatted at 2026-09-25 02:57:14 on dist-test-slave-252r"
server_key: "c4cb4a8cdf9a8682df851df1457aee5b"
server_key_iv: "2c85253b60b765b3512dfb90c8055d02"
server_key_version: "encryptionkey@0"
I20260925 02:57:14.020056 3733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-2/wal/instance:
uuid: "4a433c0a8a804f2fb2386b2d75196ab3"
format_stamp: "Formatted at 2026-09-25 02:57:14 on dist-test-slave-252r"
server_key: "c4cb4a8cdf9a8682df851df1457aee5b"
server_key_iv: "2c85253b60b765b3512dfb90c8055d02"
server_key_version: "encryptionkey@0"
I20260925 02:57:14.023452 3733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:57:14.025555 3748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:57:14.026587 3733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:57:14.026717 3733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-2/data,/tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-2/wal
uuid: "4a433c0a8a804f2fb2386b2d75196ab3"
format_stamp: "Formatted at 2026-09-25 02:57:14 on dist-test-slave-252r"
server_key: "c4cb4a8cdf9a8682df851df1457aee5b"
server_key_iv: "2c85253b60b765b3512dfb90c8055d02"
server_key_version: "encryptionkey@0"
I20260925 02:57:14.026816 3733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:57:14.042672 3733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:57:14.044934 3733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:57:14.045101 3733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:57:14.051193 3733 rpc_server.cc:307] RPC server started. Bound to: 127.12.222.252:44459
I20260925 02:57:14.051426 3800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.222.252:44459 every 8 connection(s)
I20260925 02:57:14.052053 3733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-2/data/info.pb
I20260925 02:57:14.054366 3801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:57:14.056980 13179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskORQp8j/build/debug/bin/kudu as pid 3733
I20260925 02:57:14.057111 13179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskORQp8j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304762911399-13179-0/minicluster-data/master-2/wal/instance
I20260925 02:57:14.057412 13179 external_mini_cluster.cc:1428] Setting key eee160a6f5b0aca8f5af37db6f50c471
I20260925 02:57:14.057641 3801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:57:14 dist-test-slave-252r krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305034, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.252@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
Sep 25 02:57:14 dist-test-slave-252r krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.252@KRBTEST.COM
I20260925 02:57:14.068603 3653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.059264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40138 (local address 127.12.222.254:33469)
0925 02:57:14.059383 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.059389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.059661 (+ 272us) server_negotiation.cc:407] Connection header received
0925 02:57:14.060001 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.060003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.060055 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.060149 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.060850 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.061359 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.062274 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.062484 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.062580 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:14.064902 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:14.064905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:14.064906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:14.064928 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:14.066533 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.067129 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.067132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.067133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.067201 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.067511 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.067515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.067517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.067761 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:14.067852 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:14.068341 (+ 489us) server_negotiation.cc:299] Negotiation successful
0925 02:57:14.068460 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:57:14.069360 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.059209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.254:33469 (local address 127.0.0.1:40138)
0925 02:57:14.059548 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:14.059581 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:14.059830 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:14.060195 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:14.060200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:14.060505 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:14.060710 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:14.060720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.061467 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.061472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:14.062161 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:14.062169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.062623 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.062626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:14.062801 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:14.063294 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:14.063314 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:14.064741 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:14.066638 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:14.066645 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:14.066653 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:14.067034 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:14.067278 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:14.067280 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:14.067282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:14.067401 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:14.067857 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:14.067861 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:14.068095 (+ 234us) client_negotiation.cc:769] Sending connection context
0925 02:57:14.068276 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 02:57:14.068552 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":1,"thread_start_us":122,"threads_started":1}
I20260925 02:57:14.069360 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.058501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.12.222.252:44459)
0925 02:57:14.058842 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.058862 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.058919 (+ 57us) server_negotiation.cc:407] Connection header received
0925 02:57:14.059706 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.059725 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.059965 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.060128 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.060926 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.061701 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.062520 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.062889 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.063000 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:14.065289 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:14.065293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:14.065302 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:14.065320 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:14.067043 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.067624 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.067628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.067629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.067687 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.067953 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.067956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.067957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.068171 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:14.068356 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:14.068565 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 02:57:14.068663 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":228,"thread_start_us":121,"threads_started":1}
I20260925 02:57:14.070391 3801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:57:14 dist-test-slave-252r krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305034, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.252@KRBTEST.COM for kudu/127.12.222.253@KRBTEST.COM
Sep 25 02:57:14 dist-test-slave-252r krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.254@KRBTEST.COM
I20260925 02:57:14.077721 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.070738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.253:46739 (local address 127.0.0.1:52254)
0925 02:57:14.070814 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:14.070826 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:14.070889 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:14.071154 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:14.071156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:14.071314 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:14.071419 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:14.071427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.072156 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.072158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:14.072649 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:14.072657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.073139 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.073142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:14.073228 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:14.073564 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:14.073580 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:14.074563 (+ 983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:14.076334 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:14.076338 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:14.076340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:14.076604 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:14.076882 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:14.076903 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:14.076905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:14.076994 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:14.077316 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:14.077318 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:14.077385 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 02:57:14.077464 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 02:57:14.077559 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 02:57:14.077946 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.070831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52254 (local address 127.12.222.253:46739)
0925 02:57:14.070960 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.070963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.070974 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:57:14.071026 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.071027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.071063 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.071126 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.071546 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.072065 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.072762 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.073043 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.073112 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:14.074685 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:14.074691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:14.074695 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:14.074723 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:14.076187 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.076699 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.076701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.076703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.076786 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.077086 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.077088 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.077089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.077234 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:14.077305 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:14.077564 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 02:57:14.077678 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:57:14.078481 3801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:57:14.080684 3653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.071681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40154 (local address 127.12.222.254:33469)
0925 02:57:14.072202 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.072205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.072213 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:57:14.072277 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.072279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.072312 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.072402 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.072947 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.073508 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.074174 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.074412 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.074494 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:14.077364 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:14.077376 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:14.077378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:14.077396 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:14.079281 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.079682 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.079684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.079685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.079750 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.079995 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.079996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.079997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.080118 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:14.080313 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:14.080462 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 02:57:14.080562 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464}
Sep 25 02:57:14 dist-test-slave-252r krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305034, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.252@KRBTEST.COM for kudu/127.12.222.252@KRBTEST.COM
Sep 25 02:57:14 dist-test-slave-252r krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.222.253@KRBTEST.COM
I20260925 02:57:14.086196 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.078851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44459 (local address 127.0.0.1:42840)
0925 02:57:14.078920 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:14.078929 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:14.078978 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:14.079245 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:14.079246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:14.079362 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:14.079423 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:14.079427 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.080310 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.080313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:14.080993 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:14.081007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.081462 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.081467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:14.081541 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:14.082017 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:14.082029 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:14.082988 (+ 959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:14.084694 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:14.084699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:14.084700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:14.084875 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:14.085151 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:14.085155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:14.085157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:14.085234 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:14.085563 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:14.085566 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:14.085640 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 02:57:14.085744 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 02:57:14.086016 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 02:57:14.086222 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.078970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.12.222.252:44459)
0925 02:57:14.079058 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.079062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.079074 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:57:14.079107 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.079111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.079149 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.079227 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.079539 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.080185 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.081106 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.081351 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.081446 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:14.083101 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:14.083104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:14.083106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:14.083125 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:14.084601 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.084962 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.084967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.084969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.085064 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.085330 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.085332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.085334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.085476 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:14.085572 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:14.085795 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 02:57:14.086075 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":32}
I20260925 02:57:14.089556 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.081787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52258 (local address 127.12.222.253:46739)
0925 02:57:14.081895 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.081900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.081913 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:57:14.082051 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.082054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.082088 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.082172 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.082678 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.083139 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.083823 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.084042 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.084145 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:14.086404 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:14.086407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:14.086411 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:14.086439 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:14.088173 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.088598 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.088601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.088602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.088684 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.088918 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.088921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.088922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.089096 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:14.089284 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:14.089348 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 02:57:14.089442 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:57:14.089597 3801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3: Bootstrap starting.
I20260925 02:57:14.091825 3801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:14.092345 13179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:57:14.092581 3801 log.cc:824] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:14.094173 3801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3: No bootstrap required, opened a new log
I20260925 02:57:14.093488 3625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40154
I20260925 02:57:14.096805 3801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } } peers { permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } } peers { permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } }
I20260925 02:57:14.096963 3801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a433c0a8a804f2fb2386b2d75196ab3, State: Initialized, Role: FOLLOWER
I20260925 02:57:14.097247 3801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } } peers { permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } } peers { permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } }
I20260925 02:57:14.097664 3807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } } peers { permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } } peers { permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } } }
I20260925 02:57:14.097781 3807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:14.098333 3801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 02:57:14.102571 3818 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:57:14.102631 3818 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:57:14.103036 3726 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } attempt: 3
I20260925 02:57:14.101569 3733 master_runner.cc:186] Error getting master registration for 127.12.222.254:33469: OK, instance_id { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" instance_seqno: 1790305033646637 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:57:14.104553 3733 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 25 02:57:14 dist-test-slave-252r krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305033, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.253@KRBTEST.COM for kudu/127.12.222.252@KRBTEST.COM
I20260925 02:57:14.109261 3651 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } attempt: 3
I20260925 02:57:14.111006 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.103416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42846 (local address 127.12.222.252:44459)
0925 02:57:14.103692 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.103695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.103705 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:57:14.103735 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.103737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.103766 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.103833 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.104167 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.104848 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.105563 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.105872 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.105957 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:14.107937 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:14.107939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:14.107941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:14.107959 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:14.109336 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.109921 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.109926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.109958 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.110044 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.110299 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.110302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.110303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.110458 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:14.110553 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:14.110798 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 02:57:14.110890 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260925 02:57:14.111006 3728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.103366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44459 (local address 127.0.0.1:42846)
0925 02:57:14.103502 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:14.103514 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:14.103576 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:14.103858 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:14.103860 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:14.104005 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:14.104064 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:14.104068 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.104962 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.104965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:14.105453 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:14.105461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.105969 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.105972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:14.106047 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:14.106507 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:14.106522 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:14.107803 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:14.109450 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:14.109453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:14.109455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:14.109661 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:14.110127 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:14.110129 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:14.110131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:14.110217 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:14.110547 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:14.110550 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:14.110651 (+ 101us) client_negotiation.cc:769] Sending connection context
0925 02:57:14.110740 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 02:57:14.110872 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 02:57:14.113873 3726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00: Bootstrap starting.
Sep 25 02:57:14 dist-test-slave-252r krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790305033, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.222.254@KRBTEST.COM for kudu/127.12.222.252@KRBTEST.COM
I20260925 02:57:14.116233 3726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:14.116958 3726 log.cc:824] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:14.118932 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.109571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.222.252:44459 (local address 127.0.0.1:42852)
0925 02:57:14.109685 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 02:57:14.109699 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:57:14.109765 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:57:14.110283 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:57:14.110286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:57:14.110453 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:57:14.110544 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:14.110550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.111978 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.111981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:14.112720 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:57:14.112731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.113293 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.113297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:57:14.113382 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:14.113794 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:57:14.113807 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:57:14.115060 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:57:14.117619 (+ 2559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:14.117622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:14.117623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:14.117808 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:14.118074 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:57:14.118077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:57:14.118079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:57:14.118158 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:57:14.118531 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:57:14.118534 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:57:14.118612 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 02:57:14.118680 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 02:57:14.118793 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 02:57:14.119057 3726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00: No bootstrap required, opened a new log
I20260925 02:57:14.119697 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.109781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42852 (local address 127.12.222.252:44459)
0925 02:57:14.110075 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.110087 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.110099 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:57:14.110127 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.110129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.110162 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.110663 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.110740 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.111846 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.112855 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.113181 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.114281 (+ 1100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:14.115187 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:14.115190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:14.115193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:14.115214 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:14.117472 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.117894 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.117897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.117898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.117974 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.118253 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.118256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.118257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.118432 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:14.119343 (+ 911us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:14.119433 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 02:57:14.119530 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":70,"threads_started":1}
I20260925 02:57:14.122411 3651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d: Bootstrap starting.
I20260925 02:57:14.122690 3726 raft_consensus.cc:348] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } } peers { permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } } peers { permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } }
I20260925 02:57:14.122874 3726 raft_consensus.cc:729] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91f82d25dd64807a9bf9f8ca27a7f00, State: Initialized, Role: FOLLOWER
I20260925 02:57:14.123162 3726 consensus_queue.cc:260] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } } peers { permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } } peers { permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } }
I20260925 02:57:14.123587 3822 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } } peers { permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } } peers { permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } } }
I20260925 02:57:14.123736 3822 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:14.124298 3726 sys_catalog.cc:564] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:14.124518 3651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d: Neither blocks nor log segments found. Creating new log.
I20260925 02:57:14.125226 3651 log.cc:824] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:57:14.127394 3651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d: No bootstrap required, opened a new log
I20260925 02:57:14.129976 3651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } } peers { permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } } peers { permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } }
I20260925 02:57:14.130156 3651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59596ca58a994f98a4ca4a1cd2a9a71d, State: Initialized, Role: FOLLOWER
I20260925 02:57:14.130450 3651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } } peers { permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } } peers { permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } }
I20260925 02:57:14.127236 3658 master_runner.cc:186] Error getting master registration for 127.12.222.254:33469: OK, instance_id { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" instance_seqno: 1790305033646637 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:57:14.130828 3658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:57:14.131505 3651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:57:14.131551 3625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 02:57:14.131652 3625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [term 0 FOLLOWER]: Advancing to term 1
W20260925 02:57:14.132704 3834 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:57:14.132756 3834 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:57:14.132742 3625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } } peers { permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } } peers { permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } }
I20260925 02:57:14.133576 3625 leader_election.cc:288] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [CANDIDATE]: Term 1 election: Requested vote from peers a91f82d25dd64807a9bf9f8ca27a7f00 (127.12.222.253:46739), 4a433c0a8a804f2fb2386b2d75196ab3 (127.12.222.252:44459)
I20260925 02:57:14.134181 3775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4a433c0a8a804f2fb2386b2d75196ab3"
I20260925 02:57:14.134423 3775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:14.134457 3700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00"
I20260925 02:57:14.134644 3700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:57:14.135771 3700 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59596ca58a994f98a4ca4a1cd2a9a71d in term 1.
I20260925 02:57:14.136042 3775 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59596ca58a994f98a4ca4a1cd2a9a71d in term 1.
I20260925 02:57:14.136487 3601 leader_election.cc:302] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a433c0a8a804f2fb2386b2d75196ab3, 59596ca58a994f98a4ca4a1cd2a9a71d; no voters:
I20260925 02:57:14.137256 3849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } } peers { permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } } peers { permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } } }
I20260925 02:57:14.137393 3849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:14.137475 3849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:57:14.137553 3849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [term 1 LEADER]: Becoming Leader. State: Replica: 59596ca58a994f98a4ca4a1cd2a9a71d, State: Running, Role: LEADER
I20260925 02:57:14.137827 3849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } } peers { permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } } peers { permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } }
W20260925 02:57:14.139315 3847 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:57:14.139389 3847 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:57:14.139796 3583 master_runner.cc:263] No leader master found from master 59596ca58a994f98a4ca4a1cd2a9a71d
I20260925 02:57:14.139854 3583 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:57:14.142871 3849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59596ca58a994f98a4ca4a1cd2a9a71d. Latest consensus state: current_term: 1 leader_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } } peers { permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } } peers { permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } } }
I20260925 02:57:14.142968 3849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [sys.catalog]: This master's current role is: LEADER
W20260925 02:57:14.143411 3604 tablet.cc:2369] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:57:14.151898 3858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:57:14.156725 3858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:57:14.163798 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.141177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42862 (local address 127.12.222.252:44459)
0925 02:57:14.141270 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.141274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.145618 (+ 4344us) server_negotiation.cc:407] Connection header received
0925 02:57:14.146167 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.146171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.146211 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.146259 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.150710 (+ 4451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.151466 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.158592 (+ 7126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.158936 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.159003 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:14.160606 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:14.160609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:14.160612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:14.160635 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:14.162173 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.162614 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.162618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.162619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.162692 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.162946 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.162948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.162950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.163124 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:14.163238 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:14.163552 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 02:57:14.163661 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:57:14.170476 3653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.140183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40156 (local address 127.12.222.254:33469)
0925 02:57:14.140297 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.140301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.143317 (+ 3016us) server_negotiation.cc:407] Connection header received
0925 02:57:14.143684 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.143687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.143728 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.143772 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.147989 (+ 4217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.148684 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.154526 (+ 5842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.154816 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.154896 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:14.166025 (+ 11129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:14.166029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:14.166031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:14.166056 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:14.168682 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.169098 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.169101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.169103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.169176 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.169439 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.169442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.169445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.169621 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:14.169719 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:14.170209 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 02:57:14.170306 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 02:57:14.172699 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:14.140579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52272 (local address 127.12.222.253:46739)
0925 02:57:14.140667 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.140671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.143621 (+ 2950us) server_negotiation.cc:407] Connection header received
0925 02:57:14.144625 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.144628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.144675 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.144729 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.147112 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.147857 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.155019 (+ 7162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:14.155317 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:14.155394 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:14.166377 (+ 10983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:57:14.166381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:57:14.166383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:57:14.166407 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:57:14.168027 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.171187 (+ 3160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.171191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.171192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.171277 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:57:14.171569 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:57:14.171572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:57:14.171574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:57:14.171758 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:57:14.171872 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:14.172452 (+ 580us) server_negotiation.cc:299] Negotiation successful
0925 02:57:14.172568 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:57:14.174969 3700 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00 [term 1 FOLLOWER]: Refusing update from remote peer 59596ca58a994f98a4ca4a1cd2a9a71d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:14.175668 3775 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3 [term 1 FOLLOWER]: Refusing update from remote peer 59596ca58a994f98a4ca4a1cd2a9a71d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:57:14.176430 3849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:14.176836 3849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:57:14.179929 3807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59596ca58a994f98a4ca4a1cd2a9a71d. Latest consensus state: current_term: 1 leader_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } } peers { permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } } peers { permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } } }
I20260925 02:57:14.180060 3807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:14.181370 3822 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59596ca58a994f98a4ca4a1cd2a9a71d. Latest consensus state: current_term: 1 leader_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } } peers { permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } } peers { permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } } }
I20260925 02:57:14.181488 3822 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:14.182300 3835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } } peers { permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } } peers { permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } } }
I20260925 02:57:14.182410 3835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:14.183269 3807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } } peers { permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } } peers { permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } } }
I20260925 02:57:14.183369 3807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:14.183543 3859 mvcc.cc:205] Tried to move back new op lower bound from 7333089419973181440 to 7333089419845804032. Current Snapshot: MvccSnapshot[applied={T|T < 7333089419973181440}]
I20260925 02:57:14.185398 3860 mvcc.cc:205] Tried to move back new op lower bound from 7333089419973181440 to 7333089419845804032. Current Snapshot: MvccSnapshot[applied={T|T < 7333089419973181440}]
I20260925 02:57:14.186867 3858 catalog_manager.cc:1347] Generated new cluster ID: 494db6e47b794620b2d03ba4295267f9
I20260925 02:57:14.186924 3858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:57:14.205940 3849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } } peers { permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } } peers { permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } } }
I20260925 02:57:14.206053 3849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d [sys.catalog]: This master's current role is: LEADER
I20260925 02:57:14.207028 3822 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59596ca58a994f98a4ca4a1cd2a9a71d" member_type: VOTER last_known_addr { host: "127.12.222.254" port: 33469 } } peers { permanent_uuid: "a91f82d25dd64807a9bf9f8ca27a7f00" member_type: VOTER last_known_addr { host: "127.12.222.253" port: 46739 } } peers { permanent_uuid: "4a433c0a8a804f2fb2386b2d75196ab3" member_type: VOTER last_known_addr { host: "127.12.222.252" port: 44459 } } }
I20260925 02:57:14.207134 3822 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:57:14.213056 3858 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:57:14.214584 3858 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:57:14.224107 3858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59596ca58a994f98a4ca4a1cd2a9a71d: Generated new TSK 0
I20260925 02:57:14.225368 3858 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:57:14.263631 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:14.260711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40164 (local address 127.12.222.254:33469)
0925 02:57:14.260871 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.260876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.262578 (+ 1702us) server_negotiation.cc:407] Connection header received
0925 02:57:14.262773 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.262778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.262826 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.262890 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.263517 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40164: BlockingRecv error: recv got EOF from 127.0.0.1:40164 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 02:57:14.265142 3731 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:14.261160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52282 (local address 127.12.222.253:46739)
0925 02:57:14.261262 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.261268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.263983 (+ 2715us) server_negotiation.cc:407] Connection header received
0925 02:57:14.264451 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.264454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.264495 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.264538 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.265048 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52282: BlockingRecv error: recv got EOF from 127.0.0.1:52282 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 02:57:14.266074 3820 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:14.261764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.12.222.252:44459)
0925 02:57:14.261876 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.261880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.264383 (+ 2503us) server_negotiation.cc:407] Connection header received
0925 02:57:14.265445 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.265448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.265490 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.265540 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.265984 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42876: BlockingRecv error: recv got EOF from 127.0.0.1:42876 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 02:57:14.273325 3731 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:14.269102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52288 (local address 127.12.222.253:46739)
0925 02:57:14.269228 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.269232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.272494 (+ 3262us) server_negotiation.cc:407] Connection header received
0925 02:57:14.272618 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.272621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.272686 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.272765 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.273235 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52288: BlockingRecv error: recv got EOF from 127.0.0.1:52288 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 02:57:14.277959 3731 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:14.276501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.12.222.253:46739)
0925 02:57:14.276592 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.276596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.277104 (+ 508us) server_negotiation.cc:407] Connection header received
0925 02:57:14.277284 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.277300 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.277349 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.277398 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.277881 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52290: BlockingRecv error: recv got EOF from 127.0.0.1:52290 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 02:57:14.283215 3731 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:14.281861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52294 (local address 127.12.222.253:46739)
0925 02:57:14.281958 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.281962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.282453 (+ 491us) server_negotiation.cc:407] Connection header received
0925 02:57:14.282625 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.282628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.282670 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.282721 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.283144 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52294: BlockingRecv error: recv got EOF from 127.0.0.1:52294 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 02:57:14.290325 3731 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:14.288894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52310 (local address 127.12.222.253:46739)
0925 02:57:14.288995 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.288999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.289498 (+ 499us) server_negotiation.cc:407] Connection header received
0925 02:57:14.289677 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.289680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.289725 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.289775 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.290234 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52310: BlockingRecv error: recv got EOF from 127.0.0.1:52310 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 02:57:14.301177 3731 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:14.299903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52316 (local address 127.12.222.253:46739)
0925 02:57:14.299981 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.299985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.300486 (+ 501us) server_negotiation.cc:407] Connection header received
0925 02:57:14.300627 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.300629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.300665 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.300705 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.301106 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52316: BlockingRecv error: recv got EOF from 127.0.0.1:52316 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260925 02:57:14.320158 3731 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:14.319078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52328 (local address 127.12.222.253:46739)
0925 02:57:14.319175 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.319179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.319402 (+ 223us) server_negotiation.cc:407] Connection header received
0925 02:57:14.319547 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.319550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.319596 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.319676 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.320051 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52328: BlockingRecv error: recv got EOF from 127.0.0.1:52328 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 02:57:14.354561 3731 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:14.353660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.12.222.253:46739)
0925 02:57:14.353776 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.353779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.353928 (+ 149us) server_negotiation.cc:407] Connection header received
0925 02:57:14.354024 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.354026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.354077 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.354142 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.354475 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52342: BlockingRecv error: recv got EOF from 127.0.0.1:52342 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 02:57:14.421195 3731 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:14.420215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52354 (local address 127.12.222.253:46739)
0925 02:57:14.420366 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.420372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.420499 (+ 127us) server_negotiation.cc:407] Connection header received
0925 02:57:14.420591 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.420593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.420630 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.420684 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.421096 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52354: BlockingRecv error: recv got EOF from 127.0.0.1:52354 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260925 02:57:14.552333 3731 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:14.551218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52366 (local address 127.12.222.253:46739)
0925 02:57:14.551365 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.551370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.551533 (+ 163us) server_negotiation.cc:407] Connection header received
0925 02:57:14.551647 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.551649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.551697 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.551767 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.552194 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52366: BlockingRecv error: recv got EOF from 127.0.0.1:52366 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 02:57:14.811539 3731 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:57:14.810360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52378 (local address 127.12.222.253:46739)
0925 02:57:14.810515 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:14.810520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:14.810628 (+ 108us) server_negotiation.cc:407] Connection header received
0925 02:57:14.810804 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:14.810807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:14.810861 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:14.810947 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:57:14.811390 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52378: BlockingRecv error: recv got EOF from 127.0.0.1:52378 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:57:15.103742 3818 catalog_manager.cc:1259] Loaded cluster ID: 494db6e47b794620b2d03ba4295267f9
I20260925 02:57:15.103865 3818 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3: loading cluster ID for follower catalog manager: success
I20260925 02:57:15.105386 3818 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3: acquiring CA information for follower catalog manager: success
I20260925 02:57:15.106380 3818 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4a433c0a8a804f2fb2386b2d75196ab3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:57:15.133777 3834 catalog_manager.cc:1259] Loaded cluster ID: 494db6e47b794620b2d03ba4295267f9
I20260925 02:57:15.133836 3834 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00: loading cluster ID for follower catalog manager: success
I20260925 02:57:15.135195 3834 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00: acquiring CA information for follower catalog manager: success
I20260925 02:57:15.136215 3834 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a91f82d25dd64807a9bf9f8ca27a7f00: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:57:15.329794 3933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:15.326126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52386 (local address 127.12.222.253:46739)
0925 02:57:15.326394 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:15.326398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:15.326429 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:57:15.326598 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:15.326601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:15.326735 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:15.326850 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:15.327090 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:15.327612 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:15.328396 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:15.328596 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:15.328668 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:15.328909 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:15.329335 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:15.329498 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:15.329565 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 02:57:15.329651 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":88,"threads_started":1}
I20260925 02:57:15.335981 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:15.332799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42892 (local address 127.12.222.252:44459)
0925 02:57:15.333047 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:15.333050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:15.333062 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:57:15.333134 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:15.333136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:15.333235 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:15.333319 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:15.333513 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:15.334019 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:15.334711 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:15.334913 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:15.334981 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:15.335233 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:15.335574 (+ 341us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:15.335673 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:15.335737 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 02:57:15.335777 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":81,"threads_started":1}
I20260925 02:57:15.342203 3933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:15.338592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.12.222.253:46739)
0925 02:57:15.338695 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:15.338701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:15.338982 (+ 281us) server_negotiation.cc:407] Connection header received
0925 02:57:15.339141 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:15.339143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:15.339185 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:15.339253 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:15.339479 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:15.340150 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:15.341078 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:15.341405 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:15.341500 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:15.341754 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:15.341821 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:15.341923 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:15.341992 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 02:57:15.342060 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:57:15.342386 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:15.338969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42906 (local address 127.12.222.252:44459)
0925 02:57:15.339068 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:15.339072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:15.339257 (+ 185us) server_negotiation.cc:407] Connection header received
0925 02:57:15.339452 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:15.339456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:15.339505 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:15.339566 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:15.339860 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:15.340402 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:15.341307 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:15.341550 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:15.341649 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:15.341911 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:15.341986 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:15.342058 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:15.342208 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 02:57:15.342265 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:57:15.343690 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:57:15.338584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40166 (local address 127.12.222.254:33469)
0925 02:57:15.339122 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0925 02:57:15.339126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:57:15.339138 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:57:15.339228 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:57:15.339233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:57:15.339425 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:57:15.339534 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:57:15.339738 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:15.340416 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:15.342318 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:57:15.342674 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:57:15.342781 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:57:15.343057 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:57:15.343315 (+ 258us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:57:15.343409 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:57:15.343465 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 02:57:15.343525 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":54,"threads_started":1}
I20260925 02:57:15.346544 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 3583
I20260925 02:57:15.353698 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 3658
I20260925 02:57:15.359807 13179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskORQp8j/build/debug/bin/kudu with pid 3733
2026-09-25T02:57:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2942 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5872 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (272434 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST